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EXECUTIVE SUMMARY xix 


Executive Summary 


The Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV), signed into law on 
August 12, 2006, lays out the statutory requirements governing federal support for career 
and technical education (CTE) services offered within secondary schools and postsecondary 
institutions throughout the United States. Aimed at more fully developing the academic and 
technical skills of students enrolling in CTE programs, Perkins IV stipulates the required and 
permissible uses of federal funds by states and local subgrantees and the administrative and 
compliance reporting expectations for states and local subgrantees. Since its inception in 
1984, Perkins has undergone three reauthorizations; the current legislation covers a six-year 
petiod spanning fiscal years (FY) 2008-2013. 


Congress has used successive reauthorizations of Perkins to support state and local innova- 
tion, advance programmatic initiatives to improve services, and hold grantees accountable 
for the resources they receive. One of the most noteworthy changes in Perkins IV was a new 
requirement that all local subgrantees offer one or more programs of study (POS) — career 
pathways that help students make the transition from secondary to postsecondary education 
while pursuing an industry-tecognized credential, postsecondary certificate, or degree. Per- 
kins IV also introduced new accountability requirements, including adding new indicators 
and reporting requirements aligned with the Ementary and Secondary Education Act and ex- 


tending accountability and performance reporting requirements to local subgrantees. 


The purpose of this study is to examine how state grantees and local subgrantees have 
implemented Perkins IV provisions related to finance systems, accountability, and POS, 
based on surveys and interviews with state and local administrators as well as analysis of 
extant data. Data collection was conducted in 2009 through 2010, and findings in this report 
relate to the 2008—09 and 2009-10 program years. 


Research Questions and Study Methodology 


This study presents information on the implementation of Perkins IV provisions by state 
grantees and local suberantees, including local education agencies (LEAs) and institutions of 


higher education (IHEs), in regards to the research questions listed below. 


EXECUTIVE SUMMARY 


Finance Systems 


How have federal appropriations for Perkins changed over time? 
How are Perkins IV funds distributed among state-eligible agencies? 


How are eligible agencies using Perkins IV resources to support state operations and 


local program improvement efforts? 


How are Perkins IV resources allocated among LEAs and IHEs? 


Accountability 


How are states and local subgrantees implementing Perkins IV accountability re- 


quirements? 


What challenges are states and local subgrantees facing, and how are these challeng- 


es being resolved? 


Are states and local subgrantees setting and meeting performance targets, and what 


is happening when targets are not met? 


How are data used to promote program improvement and student success? 


Programs of Study 


How are states and local subgrantees developing and implementing POS? 


What challenges are states and local subgrantees facing in developing and imple- 
menting POS? 


What is the scope of student participation in POS? 


How are states and local subgrantees addressing the core elements defined by feder- 


al legislation? 


The study used a mixed methods research design that included quantitative and qualitative 


data from a variety of sources at the state and local levels. Web-based surveys were adminis- 


tered in fall 2009 to state directors responsible for overseeing Perkins implementation at the 


secondary and postsecondary levels in the 50 states and the District of Columbia. Surveys 


were also administered to local program directors in a stratified random sample of 1,993 


LEAs, 48 independent area CTE centers, and 1,006 IHEs (in this report, survey data for area 


CTE centers were combined with LEA data due to the small sample size for the CTE cen- 


ters). State and local surveys asked primarily about program implementation during the 


EXECUTIVE SUMMARY 


2008-09 program year. Fiscal data on state reservations and suballocations for the 2009-10 
program year were obtained from 50 secondary and 49 postsecondary directors. Case studies 
were conducted in six states and 18 local communities, including site visits and in-depth in- 
terviews. An expert panel was convened to review state POS materials and guidance. The re- 
search team also compiled extant data and administrative reports maintained by federal and 


state agencies and national stakeholder groups. 


It is important to note that much of the data in this report is for years when grantees were 


still in the early stages of implementing Perkins IV provisions. 


Key Findings 


Perkins legislation has gradually evolved to keep pace with changing educational conditions. 
While Congress retained many of the key provisions and statutory requirements of preceding 
legislation in Perkins IV, it introduced several new provisions. These modifications have, 
over time, expanded the scope of the legislation, tightened administrative and compliance 
expectations, and introduced new policy initiatives intended to improve the effectiveness and 


reach of CTE instruction. Key findings identified in this study are discussed below. 


Finance Systems 


Fiscal analyses addressed how Perkins appropriations have changed over time and how funds 
are used at the state level and allocated to local secondary and postsecondary subgrantees. 
With few exceptions, study analyses reproduced findings from the 2004 National Assessment of 
Vocational Education Final Report to Congress, which provided information on the implementa- 
tion of the Carl D. Perkins Vocational and Applied Technology Education Act of 1998 (Perkins Ill). 
This result is not surprising given that resource distribution formulas in Perkins IV parallel 
those in Perkins II. One notable difference relates to Tech Prep: Perkins IV offers states the 
option of merging their Tech Prep funding into their basic grant, providing greater flexibility 


in how federal resources may be spent. 


1. When adjusted for inflation, federal Perkins allotments to states have declined over the lifetime of the 


legislation. 


With each reauthorization, Congress has escalated its investment in CTH, although these in- 
creases have not kept pace with inflation. For example, overall federal appropriations in- 
creased by 13 percent between the first years of Perkins III and Perkins IV, but they declined 
when adjusted for inflation. In real 2010 dollars, federal appropriations have declined in each 
reauthorization year, falling from nearly $1.71 billion in FY 1985 to $1.35 billion in FY 2007. 
This corresponds to a 20 percent decline in federal support for CTE, or a loss of roughly 


$360 million in purchasing power over the lifetime of Perkins. In contrast, the purchasing 
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power of the appropriations for the E/ementary and Secondary Education Act (ESEA) Title I 


grants to states more than doubled during the same period. 
2. States invest a relatively greater proportion of funding in CTE at the secondary education level. 


Perkins IV provides states with considerable discretion in how they distribute their Title I 
funds across secondary and postsecondary education. On average, states allocated nearly 
two-thirds (64 percent) of their Title I funds to the secondary level in FY 2010, a proportion 
that has remained relatively constant since FY 1992. The division of funds between educa- 
tion levels typically is negotiated by state education agency representatives to support state 
education goals. A majority of state directors indicated that they did not anticipate any 
changes in the amounts allocated to the secondary and postsecondary education levels for 


the remainder of Perkins IV. 


3. The number of grants to LEAs and IHEs has remained relatively stable across Perkins III and IV, 


as has the targeting of resources to LEAs serving high-need populations and unique areas. 


Nationally, the number of grants to LEAs and IHEs were substantially unchanged between 
Perkins II and IV. Among states for which trend data on LEA grants were available, 4,388 
grants were awarded to individual LEAs and consortia in FY 2009, compared to 4,424 grants 
in FY 2001. States for which IHE trend data are available reported a slight decline, awarding 
971 grants in FY 2009, compared to 1,065 grants in FY 2001. Perkins IV also continues to 
target resources to LEAs serving high-poverty, rural, and urban areas. In particular, high- 
poverty LEAs received $53 per student in both 2001 and 2010, while the amount per student 
in low-poverty LEAs fell from $41 to $25. On a per student basis, Perkins IV funds dispro- 
portionately benefited rural and urban LEAs in 2010. These LEAs received an average of 
$53 and $45 per secondary pupil, respectively. By contrast, the funding per pupil in suburban 
LEAs was $32. 


4. States concentrate their administrative and leadership funds on providing technical assistance and profes- 


sional development to local subgrantees. 


States may set-aside a portion of their Perkins IV grant to conduct state-level activities. 
Roughly half of states (48 percent) in FY 2009 reserved the maximum amount available for 
state grant administration (i.e., up to 5 percent of their state allotment). States also may re- 
serve up to 10 percent of their federal funds for state leadership activities, such as assessing 
CTE program performance and providing technical assistance to grant recipients. The larg- 
est percentage of state leadership funds (about 20 percent of earmarked resources) were used 
to offer technical assistance to LEAs and IHEs, followed by the delivery of professional de- 
velopment activities (roughly 15 percent) and efforts to assess the performance of CTE pro- 


grams (roughly 10 percent). State directors reported that they placed the least emphasis on 
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serving individuals in state institutions, such as state correctional facilities and institutions 


serving disabled students (about 5 percent each). 


5. States are making use of the increased grant flexibility to merge their Title LI (Tech Prep) funding into 
their Title I (basic grant) allocation. 


Perkins IV permits states to merge their Tech Prep funding into their basic grant allocation. 
As of FY 2010, 27 states had taken advantage of this option. State directors cited the desire 
to incorporate Tech Prep into all CTE programs and the perceived similarity between Tech 
Prep and POS National Assessment of Vocational Education Final Report to Congress as the most 
common reasons for merging. Both secondary and postsecondary state directors also report- 
ed wishing to avoid the data burden associated with collecting new Tech Prep measures. As 
a result of states’ consolidation of their Title I and II allocations, the number of grants to 
Tech Prep consortia declined by about half (49 percent) between 2001 and 2010, falling 
from 738 to 378. 


Accountability 


While the accountability framework in Perkins IV retains flexibility for states—by allowing 
them to use recommended indicators or existing state performance measures—the legisla- 
tion expands and refines performance requitements for states and local subgrantees. In a 
change from Perkins ILI, Perkins IV identifies separate core indicators for CTE students at the 
secondary and postsecondary education levels and introduces nine Tech Prep measures. The 
legislation also asks that states align two secondary indicators with accountability measures 
contained in ESFA and establishes new expectations for collecting and reporting valid and 
reliable data. Finally, it extends requirements for continuous improvement to the local level 
by requiring subgrantees to negotiate performance targets for each indicator with their state 
agency; those falling short of their performance goals face sanctions that may include the 


loss of some or all of their grant funding. 


1. Most states and local subgrantees are working to improve the quality of their Perkins IV accountability 


SYSTCMS. 


States and local subgrantees continue to refine the systems they use to collect and report da- 
ta. Nearly all states (49 secondary, 44 postsecondary) have adopted more than one strategy to 
ensure quality control and promote the validity and reliability of locally reported data. Com- 
mon strategies included providing technical assistance or guidance on collecting and editing 
data and performing electronic error checking and desk audits of data submitted by local 
suberantees. The Department is supporting these state efforts by offering multiple opportu- 
nities for technical assistance and by providing nonregulatory guidance to support valid, reli- 


able, and comparable reporting across states. States and local subgrantees also continued to 
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refine data collecting and reporting through initiatives such as introducing common student 
identifiers across education levels and connecting Perkins IV data systems with other educa- 


tion data systems. 


2. Perkins IV and the Department’s nonregulatory guidance offer states flexibility in how they interpret 
accountability requirements, contributing to variation in data collection and reporting practices across 


states. 


The changes to the Perkins IV accountability requirements and the introduction of the De- 
partment’s nonregulatory guidance have refined expectations for data collection and report- 
ing. Both the legislation and the guidance, however, offer states the flexibility to determine 
what indicators they will use and how they will define populations and performance 
measures. This has resulted in differing state interpretations of accountability requirements. 
While many states at least consulted the nonregulatory guidance for one or more population 
and measure definitions, only nine states used the nonregulatory guidance verbatim when de- 
fining all of their secondary populations and performance measures, and only five states did 
the same for every postsecondary population and performance measure. The flexibility in- 
herent in the legislation and the nonregulatory guidance may reduce the comparability of da- 
ta across states and pose challenges to the Department in aggregating data at the national 


level. 


3. Capacity and infrastructure influence how states and local subgrantees approach data collection and re- 


porting. 


Issues such as the capacity of student information systems, access to necessary data, timing 
of data collection and reporting, cost, and legal interpretations of federal privacy laws con- 
tinue to affect the ability of some states and local subgrantees to implement accountability 
requirements. The postsecondary placement indicator is one example: while providers are 
required to report whether students are employed, in the military, or enrolled in an appren- 
ticeship program after leaving their institution, a few states did not have the legal authority to 
access state employment databases to obtain students’ labor market outcomes. More than 30 
postsecondary state directors indicated the cost of matching student records with other state 
or national databases had at least some impact on their ability to report results for this indi- 
cator. These and other differences contribute to variations in how states report Perkins IV 


performance results to the Department and may reduce data comparability across states. 


4. States have implemented performance requirements for locals, although the effect of these requirements has 
yet to be fully measured. 


Most states had existing state policies requiring subgrantees that failed to meet targets on 


one of more core indicators to implement a local improvement plan (47 secondary, 38 post- 
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secondary).! Some states also had policies for restricting or eliminating local subgrantees’ 
flexibility in expending Perkins funds (30 secondary, 22 postsecondary), while very few had 
policies to reduce or eliminate local subgrantee funding (11 secondary, 9 postsecondary). 
Most local directors were aware of the potential new performance requirements, and 65 per- 
cent of LEA directors and 78 percent of IHE directors reported that their state would re- 
quire a program improvement plan if a local subgrantee failed to meet one or more targets. 
About half of local directors (48 percent of LEAs and 52 percent of IHEs) reported that ne- 
gotiating local targets would have no impact on CTE administration and implementation, 
and more than one-quarter thought it would have a somewhat to very positive impact. 
Among LEAs and IHEs that negotiated performance targets with their states, most experi- 
enced little to no difficulty in the negotiation process for each indicator. A large proportion 
of local directors (61 percent of LEAs and 67 percent of IHEs) also reported that introduc- 
ing sanctions would have no impact on local CTE administration and implementation, alt- 
hough very few (15 percent of LEA directors and 11 percent of IHE directors) said it would 


have a somewhat to very positive impact.? 


5. States and subgrantees use data to identify programs in need of improvement and provide technical assis- 


lance. 


States, LEAs, and IHEs have made efforts to use data for continuous improvement. More 
than half of states shared statewide averages with local subgrantees (40 secondary, 29 post- 
secondary), and many shared individual performance results with local subgrantees (35 sec- 
ondary, 30 postsecondary). The majority of states used Perkins IV accountability data to 
identify programs in need of improvement. While most states also used Perkins IV data to 
provide targeted technical assistance (45 secondary, 40 postsecondary), only about one-third 
of local subgrantees thought their state had a policy of providing professional development 
to staff of underperforming programs. Nearly half of local subgrantees used Perkins data 
quite a bit or to a great extent to identify programs that need improvement (49 percent of 
LEAs and 47 percent of IHEs) or to make program funding decisions (49 percent of LEAs 
and 48 percent of IHEs). 


Programs of Study 


One of the most substantive changes introduced in Perkins IV is the requirement that all 


states offer POS that incorporate academic, career, and technical content to prepare students 


! At the time of this study, not all states had enough information about their local providers’ 2008-09 
performance to determine how many providers were required to submit a plan for 2009-10. 

? The surveys did not provide further insight into local subgrantees’ reasons for reporting that sanc- 
tions would have no effect or a positive effect. It is possible, however, that the timing of the survey 
played a part. Local directors responded to the survey in late 2009, at the same time they were submit- 
ting the first full year of Perkins IV data. They had not yet had extensive experience with new state 
policies regarding sanctions for failing to meet negotiated targets. 
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to make successful transitions to postsecondary education and the workplace. All LEAs and 


IHEs must offer one or more POS that 


Incorporate secondary and postsecondary education elements; 


e Include coherent, rigorous content aligned with challenging academic standards and 
relevant career and technical content in a coordinated, nonduplicative progression 
of courses that align secondary education with postsecondary education to ade- 


quately prepare students for success in postsecondary education; 


e 6Offer the opportunity for secondary education students to participate in dual or 
concurrent enrollment programs or other activities to acquire postsecondary educa- 


tion credits; and 


e Lead to an industry-recognized credential or certificate at the postsecondary level or 


an associate’s or bachelor’s degree. 


In January 2010, the Department issued a POS Design Framework (Framework) that describes 
10 components considered essential to creating and implementing high-quality comprehen- 
sive POS. Released in the third fiscal year of the legislation, findings within this report assess 
the extent to which POS created by state agencies and local subgrantees address the 10 


Framework components, as well as how states are monitoring and evaluating POS. 


1. States and local subgrantees are developing and implementing POS, although the characteristics and 
quality of these programs vary. 


All secondary and postsecondary subgrantees reported that they were offering one or more 
POS, although study data revealed considerable variation in their structure and quality across 
states and, in some instances, among local suberantees within states. This variability is a re- 
sult of the flexibility that states had in implementing statutory provisions: most state ditec- 
tors reported that their state neither created legislation nor developed administrative 
guidance to support LEAs and IHEs in developing POS. A substantial number of state di- 
rectors (29 secondary, 20 postsecondary) reported that state approval is required for all POS, 
which provides some assurance that a minimum level of program quality is achieved among 
local subgrantees. Both secondary and postsecondary state directors reported that POS de- 


veloped locally with state guidance were the most common type of state-approved POS. 


2. States and local subgrantees face substantial challenges that complicate POS development and implemen- 


tation. 


Most state and local administrators described good faith efforts to ensure that each Perkins 


suberantee offered one or more POS. In working to develop these programs, state and local 
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directors most often cited a lack of funds, staff, and time as barriers to POS development. 
Both secondary and postsecondary state directors also cited a lack of understanding among 
local academic instructors about the purposes of POS, and they indicated that academic and 
CTE instructors often lacked sufficient time to plan programs and integrate curricula. Local 
LEA and JHE directors also reported that shortages of local CTE staff and lack of technical 
expertise were barriers to local POS development, though to a lesser extent than other fac- 
tors. Finally, staff in area CTE centers reported difficulty in establishing meaningful POS se- 
quences that included academic coursework taken by students in the 9th and 10th grades at 


their sending schools. 


3. Most states and local subgrantees are unable to quantify the number of students enrolled in POS or the 


outcomes that they achieve. 


Perkins IV neither provides a statutory definition of POS students nor requires that state 
agencies or local subgrantees report the number of students participating in POS or the out- 
comes of their involvement. As such, relatively few state or local Perkins subgrantees can 
provide accurate counts of the scope of student participation in POS or the educational out- 
comes associated with POS completion. Many states also lack the capacity to track second- 
ary students from high school through completion of a postsecondary credential, even in 
states and communities that have worked to establish secondary-postsecondary partnerships 
of opportunities for students to earn postsecondary credit while in high school. In some in- 
stances, state directors reported being able to track secondary students into selected in-state 
institutions (e.g., community colleges), but they were unable to assess student transitions into 


two- or four-year colleges located beyond state lines. 


4. Secondary and postsecondary administrators and instructors have difficulty coordinating POS develop- 


ment across levels. 


Although LEA and IHE program administrators reported that they cooperated to design 
POS, case study visits suggested that the formation of effective partnerships between the 
secondary and postsecondary education levels can be logistically challenging. Some case 
study respondents noted that teachers and faculty within high schools and colleges were of- 
ten physically isolated, which reduced their opportunities to collaborate. Other percetved 
barriers included the amount of time and training needed to support teachers and faculty in 
aligning secondary and postsecondary curricula, the processes required to develop local ar- 
ticulation agreements, and resistance from some postsecondary faculty, who considered POS 


primarily a strategy for secondary program delivery. 


5. Secondary students participating in POS often lack opportunities to earn college credit or earn an indus- 
try-recognized credential or a postsecondary degree or certificate. 
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Perkins IV suggests, but does not require, that POS offer secondary students the opportunity 
to earn college credit. Although three-quarters of secondary local directors reported that their 
district had articulation agreements in place with at least one postsecondary institution, only 
half reported that they offered dual credit opportunities in one or more of their POS. Fur- 
thermore, although required by the legislation, fewer than half of LEA directors reported that 
the POS offered led to an industry-recognized credential or a postsecondary certificate or de- 
gree, with many responding that they did not know whether this opportunity existed. 


Summary 


Perkins IV plays an important role in promoting continuous improvement in the organiza- 
tion and delivery of state and local CTE services. The financial support that Perkins IV pro- 
vides is critical to maintaining state leadership and keeping local programs and equipment 
up-to-date with the skill demands of a dynamic workplace. Accountability requirements con- 
tinue to hold states and local subgrantees responsible for achieving results. And the intro- 
duction of POS also is promoting alignment and coordination of CTE services across the 


secondary and postsecondary levels. 


Successive reauthorizations have, however, added to the legislation’s requirements at a time 
when many states, LEAs, and IHEs are struggling to maintain CTE programs and staff due 
to budget cutbacks. This combination of increased requirements and decreased resources 
has made it difficult for states and local subgrantees to implement Perkins IV’s components 
and monitor program effectiveness. While the Department has worked to improve the valid- 
ity and reliability of Perkins IV accountability data, measures and reporting approaches vary 
within and across states. This variation may be traced to flexibility inherent in the legislation, 
the nonregulatory nature of the Department’s guidance, and continuing challenges in track- 
ing students across secondary and postsecondary education levels. While POS offer a new 
framework for organizing the content and delivery of CTE coursework, Perkins IV resource 
distribution formulas are not designed to promote this initiative, and the absence of perfor- 
mance accountability expectations also prevents federal and state policymakers from as- 


sessing the potential contribution these programs may offer. 


As Congress begins its reauthorization deliberations, careful consideration should be di- 
rected to the scope and specificity of the current legislation. While policymakers have histor- 
ically used reauthorization to improve and redirect the legislation, for example, by adding 
expectations for POS development, the continued addition of new requirements complicates 
state and local administration and program management. Statutory flexibility also has, to 
date, allowed states to respond differently to grant requirements. Congress may wish to use 
reauthorization to assess the extent to which current legislative provisions and state flexibil- 


ity to respond contribute to achieving the legislation’s intended purposes. 
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Chapter 1. Introduction 


The Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV) extends a longstand- 
ing federal commitment to supporting career and technical education (CTE) services offered 
within states! at the secondary and postsecondary education levels. Signed into law on Au- 
gust 12, 2006, Perkins IV originally covered a six-year period encompassing 2008—2013.? 
Aimed at more fully developing the academic and career and technical skills of secondary 
and postsecondary students enrolling in CTE programs, the legislation details the required 
and permissible uses of federal funds and the administrative and compliance reporting ex- 


pectations of all state agency and local secondary and postsecondary grant recipients. 
This study examines the implementation of Perkins IV provisions related to three main areas: 
y P p 


1. Finance Systems—how state financing of CTE occurs and is changing over time; 


2. Adccountability—how, and to what extent, performance reporting requirements 


are promoting accountability and program improvement; and 


3. Programs of Study (POS)—how states and local subgrantees are integrating chal- 
lenging academic standards and rigorous technical content into sequenced, 
nonduplicative coursework aligned across secondary and postsecondary educa- 


tion levels. 


Historical Context 


Federal support for CTE has its origins in the Morrill Act of 1862, which required states to 
apply revenue from the sale of federally donated land to establish at least one college dedi- 
cated to instruction in agriculture and the mechanical arts. Following its reauthorization in 
1890, Congress adopted the landmark S’ith-Hughes National Vocational Education Act of 1917, 
the first congressional legislation to earmark funds for career studies and, in particular, agri- 


cultural education. Since then, Congress has continued to demonstrate its support for CTE 


| Perkins IV defines a “state” as “each of the several States of the United States, the District of Co- 
lumbia, the Commonwealth of Puerto Rico, and each outlying area” (Sec. 3). The term “state” as it is 
used in this report includes the 50 states of the United States, the District of Columbia, the Com- 
monwealth of Puerto Rico, and the outlying areas of the United States Virgin Islands, Guam, Ameri- 
can Samoa, the Commonwealth of the Northern Mariana Islands, and the Republic of Palau. 

? Congress extended the legislation to cover the 2014-15 program year. 
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by directing federal resources to CTE programs and students through successive legislative 


efforts. 


Early Support 


Congress laid the foundation for contemporary CTE fiscal and accountability policies at the 
dawn of the Great Society era with its authorization of the Vocational Education Act of 1963. 
The legislation introduced a federal-to-state allotment formula that allocated funds based on 
the number of students in different age categories within states, a variant of which is still 
employed in Perkins IV. Congress also used the 1963 legislation to abolish categorical fund- 
ing for specific CTE program areas, such as agriculture, giving states increased flexibility in 
spending federal funds. To hold states accountable, the legislation mandated that states con- 
duct follow-up studies to assess the extent to which graduates entered training-related em- 


ployment, although states were not required to share these results with the Department. 


The Vocational Education Amendments of 1968 expanded expectations for state expenditures of 
federal funds, introducing a state-to-local distribution formula that earmarked funds for dis- 
advantaged populations, out-of-school individuals seeking employment, and the handi- 
capped. The 1968 legislation also called for state advisory councils to conduct evaluations of 
program activities, again without imposing any reporting requirements. The legislation pre- 
ceding the Perkins era, the Educational Amendments of 1976, aagmented federal investment in 
CTE while maintaining existing resource distribution formulas. The 1976 legislation also re- 
quired that states evaluate whether those who had completed or left programs found train- 
ing-related employment and whether employers were satisfied with the preparation of these 


individuals. 


The Perkins Era 


Congressional passage of the Carl D. Perkins Vocational and Technical Education Act of 1984 
(Perkins I) established the legal precedent upon which more than a quarter of a century of 
federal CTE policy rests. Perkins I included an innovation affecting how state funds were to 
be expended: specifically, it set aside just over half of its resources for special populations, 
including the handicapped, the disadvantaged, adults in retraining, single parents and home- 
makers, and the incarcerated. Remaining resources were directed to program improvement. 
Perkins I also laid the groundwork for a formal accountability framework, requiring states to 
develop measures to assess the effectiveness of vocational education programs in meeting 


state labor market needs and in supplying workplace competencies desired by employers. 


The Carl D. Perkins Vocational and Applied Technology Education Act of 1990 (Perkins Il) intro- 
duced a substantive policy shift in federal support for CTE. With respect to finances, Perkins 
II reformulated the state-to-local distribution formula, requiring that three-fourths of funds 
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be allocated as a basic grant predicated on the number of persons enrolling in eligible institu- 
tions. At the secondary level, federal funds were distributed based on the number of youth in 
participating local agencies, with 70 percent based on the number living in poverty, 20 per- 
cent on the number who were handicapped, and 10 percent on the total number enrolled. 
Postsecondary funds were allocated based on the number of Pell Grant recipients and recip- 
ients of assistance from the Bureau of Indian Affairs, the formula that continues to be used 


today. 


Equally noteworthy in Perkins II was the fact that it specified a set of performance accounta- 
bility requirements. The legislation directed states to create measures of learning and compe- 
tency gains—including those related to basic and advanced academic skills—and to adopt 
one of more measures of performance related to technical competency attainment; job or 
work skills enhancement; retention in schooling; or placement into additional training, edu- 
cation, military service, or employment. Although Perkins I did not require states to report 
outcome data on these measures to the Department, it held local recipients responsible for 
conducting program evaluations and implementing improvement plans if they failed to make 


substantial progress toward meeting identified standards. 


Perkins II also is distinctive for introducing Tech Prep? as a demonstration grant program. 
Tech Prep was first introduced in high schools in the early 1980s as a locally driven im- 
provement strategy that offered a combined secondary and postsecondary program leading 
to the award of an associate’s degree or a certificate. Congress used Perkins II to support 
consortia of secondary and postsecondary agencies in developing and operating Tech Prep 
programs that, because of their focus on coordinating instruction across education levels, 
serve as a precursor to the programs of study (POS) framework introduced in the Perkins IV 


legislation. 


The Carl D. Perkins Vocational and Applied Technology Education Act of 1998 (Perkins IID) formal- 
ized the funding and accountability criteria upon which Perkins IV is based. Specifically, Per- 
kins III increased the percentage of local resources dedicated to the basic grant (from 75 to 
85 percent) and introduced a reserve fund to enhance state flexibility in allocating federal 
CTE funds. Perkins III also eliminated previous set-asides for some special populations and 
for state gender equity coordinators, and it voided requirements that local agencies direct 


funds to schools and programs with the greatest concentration of special populations. 


Perkins III further expanded state accountability requirements, introducing a set of four core 
P q , 8 

performance indicators on which all states were required to report. States failing to achieve 

performance targets negotiated with the Department faced sanctions, beginning with devel- 


oping a program improvement plan and extending to the loss of some or all of their federal 


3 “Tech Prep” refers to the content of tech prep programs described in Sec. 3(32) of Perkins IV. 
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allotment. In keeping with provisions introduced in Perkins IT, Perkins III held local subgrant- 
ees responsible for making substantial progress in achieving state-established performance 


levels, with those falling short directed to develop a program improvement plan. 


Perkins IV: Federal CTE Policy Today 


Current federal CTE policy is governed by the provisions of Perkins IV. Although Perkins IV 
formally went into effect in the 2007-08 program year,* many policies governing state and 
local accountability and POS implementation were delayed a year to allow states to develop a 
five-year plan for implementing legislative requirements. During the one-year transition peri- 
od, state administrators consulted with local subgrantees, educators, students and parents, 
institutions of higher education, representatives of business and industry, and other stake- 
holders to identify any changes needed to state policies, programs, or accountability systems, 
as they began to implement these changes for the 2008-09 program year. States submitted 
their five-year state plans for Perkins IV on April 1, 2008. 


Although Perkins IV introduces some innovations, consistency with past legislation is an 
overarching theme. This study focuses on assessing state and local implementation of Perkins 


IV in three key areas, as described below. 


Finance systems 
Perkins IV continues most of the fiscal policies contained in Perkins III. Congress primarily 
fine-tuned formula operations by, for example, introducing new allotment procedures for 
years in which new federal funds are appropriated. The most substantive change in fiscal 
policy introduced in Perkins IV relates to Tech Prep: states now can consolidate all or a por- 
tion of their Tech Prep allocation with their basic grant. States exercising this option must 
treat consolidated funds as they would other funds allocated as part of their basic grant. This 
new provision has increased state flexibility in allocating funds to secondary eligible recipi- 
ents and postsecondary eligible institutions. 5 For the purposes of this report, the terms LEA 
and IHE are used to refer to all local subgrantees at the secondary and postsecondary educa- 


tion levels that are eligible recipients of Perkins IV funding. 


4 A program year begins on July 1 and ends on June 30 of the following calendar year. The 2007-08 
program year began on July 1, 2007, and ended on June 30, 2008. 

> Perkins IV Sec. 3. Secondary eligible recipients include local education agencies, charter schools op- 
erating as a local eligible recipient, an area career and technical school, an educational service agency, 
or a consortium. A postsecondary eligible institution may include a public or nonprofit private institu- 
tions of higher education that offers CTE courses that lead to technical skill proficiency, and industry 
recognized credential, a certificate, or a degree; and postsecondary subgrantees also include a small 
proportion of middle schools and adult programs, a local education agency providing education at the 
postsecondary level, an area career and technical education school providing education at the postsec- 
ondary level; a postsecondary institution operated by the Bureau of Indian Affairs or operated by or 
on behalf of an Indian tribe; an educational service agency; or a consortium. 
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Accountability 
Perkins IV continues to hold state eligible agencies® responsible for collecting and reporting 
performance data but asks for a higher level of data quality. In a break from earlier legisla- 
tion, federal policymakers crafted a separate set of core indicators for secondary and post- 
secondary education and introduced new indicators for Tech Prep programs, previously not 
subject to measurement. Only states that elect to maintain a separate Tech Prep funding al- 


location are accountable for the new Tech Prep indicators. 


The legislation also incorporates two core indicators from the Elementary and Secondary Educa- 
tion Act of 2002 (ESEA) to link CTE student performance to academic performance assess- 
ments and high school graduation measures used for all secondary students. Finally, 

Perkins IV extends the expectations for reporting and continuous improvement to the local 
level, requiring that secondary and postsecondary subgrantees negotiate performance goals 
for each indicator with the state. Those that fall short of their negotiated targets face pro- 
gressive sanctions that begin with developing a program improvement plan and may culmi- 


nate in the loss of some or all of their Perkins IV funds. 


Programs of study 
Perkins IV introduces the requirement that all local recipients of its funds offer one or more 
POS to link secondary and postsecondary CTE. Programs of study can be defined as a se- 
quenced, nonduplicative progression of courses that intentionally connect high school and 
postsecondary CTE curricula and lead to an industry-recognized credential or certificate, or 
an associate’s or bachelor’s degree. Curriculum and instruction are standards-based and em- 
phasize the linkages between rigorous academic and technical content. Where appropriate, 
students also can earn college credits while still enrolled in high school. Students completing 
the secondary component of a POS can transition seamlessly into an articulated postsecond- 
ary program without the need for remedial education or duplicating coursework that they 
completed in high school. Although local subgrantees are required to make POS available, 
they are neither accountable for student performance in these programs nor asked to collect 


data on student participation or outcomes. 


Research Questions 


This study uses a complex research design to gather information on how states, LEAs, and 


IHEs organize, administer, and deliver CTE services. Study activities call for evaluating three 


6 According to Perkins IV, an “eligible agency” is a state board designated or created consistent with 
state law as the sole state agency responsible for the administration of career and technical education 
in the state or for the supervision of the administration of career and technical education in the state 
(Sec. 3(12)). 
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aspects of state and local implementation of programs funded by Perkins IV, with the goal of 


answering the following research questions, among others: 


e Finance Systems 
o How have federal appropriations for Perkins changed over time? 
o How are Perkins IV funds distributed among state-eligible agencies? 


o How are eligible agencies using Perkins IV resources to support state opera- 


tions and local program improvement efforts? 
o How are Perkins IV resources allocated among LEAs and IHEs? 
e Accountability 


o How are states and local subgrantees implementing Perkins IV accountabil- 


ity requirements? 


o What challenges are states and local subgrantees facing, and how are these 


challenges being resolved? 


o Are states and local subgrantees setting and meeting performance targets, 


and what is happening when targets are not met? 
o How are data used to promote program improvement and student success? 
e ©Programs of Study 
o How are states and local subgrantees developing and implementing POS? 


o What challenges are states and local subgrantees facing in developing and 
implementing POS? 


o What is the scope of student participation in POS? 


o How are states and local subgrantees addressing the core elements defined 


by federal legislation? 


Data Collection 


Answering the study research questions required assembling a substantial amount of quanti- 
tative and qualitative data, using a mixed methods research approach. Study activities includ- 
ed collecting original source data and compiling extant data and administrative reports— 

maintained by federal and state agencies or produced by CTE researchers, professional asso- 


ciations, and national stakeholder groups. The following section provides a brief summary of 
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the data collection approaches used for this study; a detailed description of the study meth- 
odology is presented in Appendix A. 


Original Data Sources 


The research team constructed a set of data collection tools to address the study research 
questions. This task included formulating web-based surveys to collect feedback from state 
CTE directors and LEAs and IHEs; collecting fiscal allocation data from states; conducting 
case study site visits to a subset of representative states; and conducting expert panel reviews 
of state POS materials and guidance. Protocols for structuring case design visits and expert 


panel reviews were pilot-tested with state and local CTE administrators prior to their use. 


State director surveys 

Each state designates a state director of CTE with primary oversight of the federal grant. 
Typically, this person is located at either a state secondary or postsecondary education agen- 
cy, board, or commission, though other administrative structures exist. To ensure the collec- 
tion of comprehensive data, researchers developed and administered separate surveys to the 
state director and the person charged with CTE oversight at the counterpart secondary or 
postsecondary education agency. Surveys were administered to a total of 57 secondary and 
57 postsecondary state directors.’ Study findings are based on responses obtained from 51 


secondary and 48 postsecondary state directors. 


LEA and IHE surveys 


To identify a random, representative sample of survey participants, the researchers construct- 
ed a stratified sampling frame from two National Center for Education Statistics (NCES) da- 
tasets: the Common Core of Data (CCD) and the Integrated Postsecondary Education Data 
System (IPEDS). They then administered surveys to local directors at a random sample of 
1,993 LEAs and 1,006 IHEs, as well as 48 area CTE centers that operated independent of a 
school district.*.° Completed surveys were received from 77 percent of LEAs, 91 percent of 


THEs, and 93 percent of area CTE centers. 


7 The term “state director” as it is used in this report refers to the individuals at the secondary and 
postsecondary education levels who are charged with oversight of federal grant resources and who te- 
sponded to the secondary and postsecondary state director surveys. 

8 The term “local director” as it is used in this report refers to the individuals at LEAs, IHEs, and area 
CTE centers who responded to the LEA and THE surveys. 

° Area CTE centers provide CTE instruction to students who receive all or most of their academic in- 
struction at their home high school. An area CTE center often serves multiple high schools within 
multiple school districts. 
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State fiscal allocation survey 
Federal resources are allotted to states using a formula specified in Perkins IV. A portion of 
these funds are used by state eligible agencies to administer the grant and offer technical as- 
sistance and program support; remaining funds are distributed among participating LEAs 
and IHEs. To assess these state-to-local 


formula allocations, researchers asked state 


directors to submit their Title I and Title I Grant Compo nents 


fiscal allocation data for LEAs and IHEs in 
the 2009-10 program year. Researchers also 


requested information on consortia mem- Perkins IV resources are allocated to states for 
bership and state-level allocations of leader- different purposes, as follows: 

ship and administrative funds. Fiscal data Title I—CTE Assistance to the States 

were obtained from 50 secondary and 49 Details the formula for distributing federal re- 


sources among states for state administrative 
uses and for state formula allocations to local 
providers as basic grants. 


postsecondary state directors. 


Case study site visits 


Researchers conducted case study visits to We uGen Sao belies tet 


Describes the allotment and uses of federal 


funds to support Tech Prep programs offered 
sent the continuum of state approaches to by local providers. 


six states. The states were selected to repre- 


implementing Perkins IV legislative provi- : ae 
Title III—General Provisions 


Summarizes federal and state administrative 
terviews with secondary and postsecondary provisions governing the uses of funding. 


state directors and administrative staff, and 


sions. Site visitors conducted in-depth in- 


visited three randomly selected local com- 
munities (one urban, one suburban, and one rural) to collect data from LEA and IHE ad- 
ministrators, faculty, and staff. Visits also included meetings with business, industry, and la- 


bor representatives, where feasible. 


Expert panel reviews 
The research team convened and facilitated virtual meetings of a 10-member expert panel to 
assess the usefulness of state CTE guidance and technical assistance materials related to POS 
design and implementation. Panel members reviewed state materials submitted by state di- 
rectors as part of the survey effort, along with other publicly available documents posted 
online. Materials were obtained for the 50 states and the District of Columbia. A list of ex- 


pert panel members is included in Appendix B. 
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Extant Data Sources 


Researchers conducted an exhaustive search of the literature to identify existing data and 
published reports documenting efforts by the Department, state education agencies, and lo- 


cal subgrantees to implement Perkins IV’ finance, accountability, and POS provisions. 


Department databases 
Each year, states report financial and accountability data to the Department to be used for 
Perkins IV compliance and monitoring purposes. Study researchers obtained permission 
from Department staff to access relevant administrative databases to identify state uses of 


federal resources. 


Additional resources 
Federal and state education agencies annually release information on the design, implementa- 
tion, and administration of secondary and postsecondary CTE programs, including those 
funded with Perkins IV resources. Researchers accessed publicly available data and reports 
posted on Department websites, along with those maintained by state secondary and post- 
secondary education agencies, to identify information relevant to the study. They also ac- 
cessed websites maintained by public and private CTE stakeholders and associations, and 
searched relevant literature for supplemental information about the design and implementa- 
tion of Perkins IV. Finally, they accessed state five-year Perkins IV plans and annual plan up- 


dates to assess ongoing grant administration. 


Organization of the Report 


This report summarizes data and information collected as part of the study. Each chapter 
opens with a summary of key findings. Throughout the report, and where appropriate, data 
from multiple sources have been integrated into the text to support and illustrate key study 
findings. For example, observations and feedback collected through case study site visits re- 
inforce findings distilled from the state CTE director and LEA and IHE surveys. Similarly, 
data from the state fiscal allocation survey are supplemented with information from the fed- 
eral database review to provide additional information on state uses of funds for administra- 


tive purposes. 


Chapter 2 provides a summary of Perkins IV fiscal policies, documenting federal allotments 
of funds to states, uses of funds by states for administration and leadership, and state alloca- 
tions to local providers. Where appropriate, trend data illustrate changes in states’ receipt of 


federal funds and how resources are targeted to student subpopulations and programs. 
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Chapter 3 presents a detailed discussion of federal efforts to improve the collection of state 
accountability data and documents issues affecting the comparability, reliability, and validity 
of performance data. The chapter reviews the changes in accountability under Perkins IV; 
outlines challenges that states face in collecting and reporting data; and discusses how states 
and subgrantees negotiate performance targets, address difficulties in meeting targets, and 


use performance results to improve programs. 


Chapter 4 provides information on the development of POS—one of the most substantive 
changes introduced in Perkins IV—and issues related to local implementation. The chapter 
describes how states, LEAs, and IHEs design POS, the scale of these programs, the strate- 
gies states used to promote their implementation, and the challenges encountered in the de- 


sign and implementation process. 


The report closes with a summary of key findings found across the three topical areas and is- 
sues deserving continued attention. Appendix A describes the study methodology and tech- 
nical documentation associated with data collection and includes copies of the protocols 
used to collect survey data and to conduct case study interviews. Appendix B summarizes 
the expert panel review of state POS materials and guidance. Finally, Appendix C presents a 
cross case analysis of information collected during case study visits to six states, and three 
local partnerships within each state, selected as representative of the range of approaches 


states are using to develop and implement POS. 
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Chapter 2. Finance Systems 


The Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV) details the funding 
distribution formulas and criteria used to allot the nearly $1.3 billion that the federal gov- 
ernment invests annually in secondary and postsecondary career and technical education 
(CTE). This chapter examines the flow of Perkins IV basic grant (Title I) and Tech Prep (Ti- 
tle II) funds from the federal level to state eligible agencies, and from eligible agencies to lo- 
cal education agencies (LEAs) and institutions of higher education (IHEs). 


The analyses in this chapter seek to answer the following research questions: 


e How have federal appropriations for Perkins changed over time? 
e How are Perkins IV funds distributed among state-eligible agencies? 


e How are eligible agencies using Perkins IV tesoutces to support state operations 


and local program improvement efforts? 
e How are Perkins IV resources allocated among LEAs and IHEs? 


The chapter opens with an assessment of how federal fiscal priorities and distribution for- 
mulas for CTE changed with Perkins IV in 2006 and the implications of these changes for 
states and local providers. This discussion is followed by an analysis of how states allocate 
federal resources in support of secondary and postsecondary education programs. The re- 
view of state grants is divided into two parts. The first addresses how funds are spent at the 
state level for state administration and state leadership, and the amounts dedicated to ser- 
vices for individuals in institutions and preparing for nontraditional occupations. The second 
addresses how funds are allocated to LEAs and IHEs through the use of the reserve fund 
and the statutory formula that governs the disbursement of remaining Title I funds. The 
analysis also addresses local spending for required and permissive uses and how local CTE 


administrators determine how funds are spent. 


The final sections address the flow of Tech Prep funds from the federal to state and local 
levels and the option states have for consolidating Title I and II funds. The discussion closes 
with an examination of how Tech Prep funds are allocated by states to consortia of LEAs 


and IHEs and how recipients use these resources. 
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Key 


This chapter is based on survey data submitted by state directors and local directors, fiscal 
data obtained from state directors and staff, site visits with state and local directors and CTE 


staff, and information retrieved from federal administrative databases. 


Findings 


1. When adjusted for inflation, federal Perkins grants to states have declined over the lifetime of the legisla- 


ton. 


Federal appropriations for CTE, in real 2010 dollars, have declined in each reauthorization 
year, falling from nearly $1.71 billion in FY 1985 to $1.36 billion in FY 2007. This represents 
a 20 percent decline in federal support for CTE, a loss of roughly $350 million in purchasing 
power, between the first year of the Car/ D. Perkins Vocational and Appled Technology Education 
Act of 1985 (Perkins I) and the most recent reauthorization. 


2. States invest a greater proportion of Perkins funding at the secondary education level. 


On average, states allocated about 64 percent of their Title I funds to the secondary level in 
FY 2010, a proportion remaining relatively constant since FY 1992. The average percentage 
of Title I resources allocated to the secondary level between FY 2001 and FY 2010 differed 
by 5 percent or less in 33 of the 44 states for which multi-year data are available. Among the 
eleven states with percentage changes greater than 5 percent, five decreased the proportion 
of resources flowing to the secondary level, resulting in a more balanced split across the sec- 


ondary and postsecondary education levels. 


3. The number of LEA subgrantees has remained relatively stable across Perkins II and IV, as has the 
targeting of resources to LELAS serving high-need populations and rural areas. The number of IHE and 
other postsecondary subgrantees declined by about 9 percent between FY 2001 and FY 2010. 


Nationally, the number of LEA subgrantees was substantially unchanged between Perkins II 
and Perkins IV. Among the 39 states for which trend data were available, there were 4,388 
LEAs and consortia subgrantees in FY 2010, compared to 4,424 subgrantees in FY 2001, a 
difference of less than 1 percent. The number of local postsecondary grants declined by 
about 9 percent over the same time period; states reported awarding 971 grants in FY 2010 
and 1,065 grants in FY 2001. As in Perkins II, LEAs serving high-poverty, rural, and urban 
areas received relatively larger proportions of Perkins IV resources. High-poverty LEAs tre- 
ceived $53 per student in both FY 2001 and FY 2010, while the amount per student in low- 
poverty LEAs fell from $41 to $25. On a per student basis, Perkins IV funds disproportion- 
ately benefited rural and urban LEAs in FY 2010. These LEAs received an average of $53 
and $45 per secondary pupil, respectively. By contrast, the funding per pupil in suburban 
LEAs was $32. 
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4. States concentrate their administrative and leadership funds on providing technical assistance and profes- 


sional development to local subgrantees. 


A portion of each state’s Perkins IV grant may be set-aside for state-level activities. Nearly 
half of states (47 percent) in FY 2009 reserved the maximum amount allowed for state grant 
administration. States also may reserve up to 10 percent of their federal funds for state lead- 
ership activities, such as assessing CTE program performance and providing technical assis- 
tance to subgrantees. State directors reported that the largest percentage of state leadership 
funds (about 20 percent) supported technical assistance to LEAs and IHEs, followed by the 
delivery of professional development activities (roughly 15 percent) and efforts to assess the 
performance of CTE programs (roughly 10 percent). Reflecting legislated limits, the lowest 
percentage was for serving individuals in state institutions, such as state correctional facilities 


and institutions serving disabled students (about 5 percent of leadership funds). 


5. States are making use of the increased grant flexibility to merge their Title II (Tech Prep) and Title I 
(basic grant) funds. 


As of FY 2010, a total of 27 states had merged their Title I and II funds, with one state con- 
solidating only a portion of its resources. The most common reasons for merging cited by 
secondary state directors were the desire to incorporate Tech Prep into all CTE programs, 
followed by the similarity between Tech Prep and programs of study (POS). A further incen- 
tive, mentioned by both secondary and postsecondary state directors, was that merging funds 


allowed states to avoid the data burden associated with collecting new Title II measures. 
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The Federal Investment in CTE 


Federal funding for Perkins IV has declined since the legislation’s passage in 2006; appropria- 
tions fell by roughly 2 percent between FY 2007 and FY 2008, declining from $1.30 to $1.27 
billion. Congressional appropriations have since remained constant at $1.27 billion annually 
in FY 2009 and FY 2010 (U.S. Department of Education 2010, 2010a). Historically, howev- 
er, the dollar amount of Perkins grants to states and territories has increased with each legisla- 
tive reauthorization (Exhibit 2.1).! In nominal dollars, federal appropriations for CTE 
climbed by roughly 20 percent between 1985 and 1991 (the first years of Perkins I and Perkins 
ID), rising from $0.84 billion in FY 1985 to $1.01 billion in FY 1991.2 Following the reauthor- 
ization of Perkins II, first-year appropriations rose by 14 percent to $1.15 billion by FY 1999. 
Finally, congressional authorizations for the first year of Perkins IV climbed by about 13 per- 
cent from FY 1999 to FY 2007, increasing to about $1.3 billion. 


Although federal appropriations for CTE rose in nominal dollars from FY 1985 to FY 2007, 
appropriations declined over this period when adjusted for inflation. In real 2010 dollars, 
federal appropriations for CTE fell from nearly $1.71 billion in FY 1985 (Perkins I) to $1.36 
billion in FY 2007 (Perkins IV) (Exhibit 2.1). This represents a 20 percent decline in federal 
support for CTE, and is equivalent to a loss of roughly $350 million in purchasing power be- 
tween the first years of Perkins I and Perkins IV. 


! The amounts cited here are based on federal appropriations in the first fiscal years of Perkins I 
through Perkins IV; Congress determines federal appropriations for Perkins in the fiscal year preceding 
the year in which the funds are expended by states. For example, Congress made the first appropria- 
tions for Perkins IV in FY 2007 for use by states during FY 2008. 

? Appropriations for CTE are allocated through Perkins programs. Nominal dollars are amounts unad- 
justed for inflation. Real dollars are amounts adjusted for inflation with reference to a particular year. 
The real dollars reported here were calculated using the inflation calculator maintained by the U.S. Bu- 
reau of Labor Statistics (http://www.bls.gov/data/inflation_calculator.htm) (accessed January 20, 
2011). 
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Exhibit 2.1. 
Federal fiscal year appropriations for CTE in nominal and real 2010 dollars (in billions of 
dollars) in the first years of Perkins Ito Perkins IV: Fiscal Years 1985, 1991, 1999, and 2007 
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Exhibit reads: In nominal dollars, the federal appropriation for CTE was $1.30 billion in FY 2007 and $0.84 billion in FY 1985. In real 
2010 dollars, the federal appropriation for CTE was $1.36 billion in FY 2007 and $1.71 billion in FY 1985. 

NOTE: Appropriations for CTE are allocated through Perkins programs. 

SOURCE: U.S. Department of Education Budget History Tables, FY 1980-FY 2009, 
http://www2.ed.gov/about/overview/budget/history/edhistory.pdf (accessed January 12, 2011); and Real dollar calculations: Bureau of 
Labor Statistics, Inflation Calculator, http://data.bls.gov/cgi-bin/cpicalc.pl (accessed January 20, 2011). 


By contrast, federal appropriations for the U.S. Department of Education increased from 
18.9 to 67.1 billion dollars in nominal terms, and from 38.5 to 70.9 billion in real 2010 dol- 
lars, an increase in purchasing power of about 84 percent.? Funding for some major pro- 
grams has increased accordingly. For example, appropriations for the Ementary and Secondary 
Education Act (ESEA)* Title I grants to LEAs increased in both nominal and real 2010 dol- 
lars over the same period (Exhibit 2.2). The nominal increase in funds from about $3.2 to 
$12.8 billion from FY 1985 to FY 2007 is equivalent to an increase in purchasing power of 
about $7 billion in real dollars, or about 108 percent. 


3 Education Department Budget History Table FY 1980-FY 2009, retrieved September 19, 2010, 
from http://www2.ed.gov/about/overview/budget/history/edhistory.pdf. 
4P.L. 89-10, 79 Stat. 27. 
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Exhibit 2.2. 
Federal appropriations for Elementary and Secondary Education Act (ESEA) Title I grants 
to LEAs (in billions of dollars): Fiscal Years 1985, 1991, 1999, and 2007 
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Exhibit reads: In nominal dollars, the federal appropriation for ESEA ‘Title I grants to LEAs was $12.84 billion in FY 2007 and $3.20 
billion in FY 1985. In real 2010 dollars, the federal appropriation for this program was $6.48 billion in FY 1985 and $13.50 billion in FY 
2007. 

SOURCE: U.S. Department of Education Budget History Tables, FY 1980-FY 2009, 
http://www2.ed.gov/about/overview/budget/history/edhistory.pdf (accessed January 12, 2011); and Real dollar calculations: Bureau of 
Labor Statistics, Inflation Calculator, http://data.bls.gov/cgi-bin/cpicalc.pl (accessed January 20, 2011). 


Title | National Activities and Programs 


Most federal CTE funds are allotted through Perkins Title I programs. In FY 2007, Title I 
funding was $1.18 billion, or about 91 percent of the total $1.30 billion appropriated. The 
appropriation for Title II programs was $105 million, or about 8 percent of the total.> In ad- 
dition to funding state grants, Perkins IV Title I funds provide for a number of national ac- 
tivities aimed at improving CTE quality and effectiveness. These funds support research, 
development, and evaluation through a national research center and the National Assess- 
ment of Career and Technical Education (NACTE) study. In FY 2007, about $10.0 million 
was appropriated for these programs. The amount fell to $7.86 million in FY 2008, and re- 
mained constant at that level in FY 2009 and FY 2010.7 


5 Education Department Budget History Table FY 1980-FY 2009, retrieved September 19, 2010, 
from http://www2.ed.gov/about/overview/budget/history/sthistbypr01 to08.pdf. 

6 Sec. 114. 

7 FY 2011 Department of Education Justifications of Appropriation Estimates to the Congress, re- 
trieved February 10, 2010, from 

http://www2.ed.gov/about/overview/budget/budget1 1/justifications/index.html. 
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In keeping with preceding legislation, Congress has also reserved a portion of Title I re- 
sources as set-asides for special purposes; these amounts are deducted before funds are dis- 
tributed to states. Perkins IV introduced two changes to these set-asides.* The first ended the 
eligibility of Micronesia and the Marshall Islands for funds, reducing funds to outlying areas 
from 0.2 to 0.13 of the Perkins Title I appropriation.® The second change eliminated the state 
incentive grant program introduced in Perkins II (1998) and maintained from FY 2000 to FY 
2003. This program provided financial rewards for states meeting their negotiated perfor- 
mance targets for Perkins III and Titles I and H of the Workforce Investment Act (WILA).'° 


In addition to the assistance for outlying areas, Congress also provides sepatate appropria- 


tion for three other Title I programs: 


e Native American Programs (Sec. 116)—to provide grants to Indian tribes, tribal 
organizations, and Alaska Native groups to carry out CTE programs consistent 
with the legislation. A portion of the overall appropriation under this section is 
reserved for grants to community-based organizations serving and representing 


Native Hawaiians for the same purpose. 


e = Tribally Controlled Postsecondary CTE Institutions (Sec. 117)—to provide 
grants to support CTE programs for Indian students and offset institutional 


support costs. 


e Occupational and Employment Information (Sec. 118)—to provide grants to 
states for technical assistance and the dissemination of information, products, 


and services to assist states in carrying out the legislation. 


Funds for Title I of the legislation, covering Tech Prep education, are allotted through a 
separate appropriation, with state grant eligibility estimated using the same distribution for- 
mula used to award Title I funds. Title II funds to states and distribution trends are dis- 


cussed in a later section. 


Title | State Grants 


After funds reserved for special purposes and the programs are deducted, nearly all remain- 


ing Title I funds—treferred to as a “basic grant”’—are allotted by formula to the 50 states and 


8 Sec. 111(a)(1). 

° Outlying areas and their grant eligibilities and amounts are stipulated in Perkins IV (Sec. 3(21) and 
Sec. 115). These areas include American Samoa, Guam, the Commonwealth of the Northern Mariana 
Islands, and Palau. Although defined as an outlying area, the U.S. Virgin Islands receive Perkins IV 
funds through the federal state grant allocation formula. 

10 PL. 105-220, 112 Stat. 936. 
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the District of Columbia, Puerto Rico, and the U.S. Virgin Islands." State grants are based 
on states’ relative share of the national population across four age groups and the state’s per 
capita income, averaged over three years, relative to the national average. Generally, the in- 
come adjustment means that a state with a lower average per capita income will recetve more 


funding per student than one with higher relative wealth. 


Perkins IV also places several conditions on the Title I grants received by states, and initial 
amount calculations are adjusted until these conditions are met.!2' Two provisions place a 
floor on the total amount that states can receive. According to the special rule known as the 
small state minimum, no state can receive less than 0.5 percent of basic grant funds (excluding 
set-asides). In FY 2009, the small state minimum was $5.7 million. Further, the ho/d-harmless 
provision prohibits states from receiving less than their FY 1998 grant. Another provision 
caps grant amounts at no more than 150 percent of their prior fiscal year grant, or 150 per- 
cent of the national per pupil payment of Perkins funds multiplied by the number of individ- 
uals counted in the state’s Perkins formula, whichever is less.'3 In practice, these limits have 
capped grant amounts for some states qualifying for the small state minimum grant at an 
amount less than 0.5 percent of basic grant funds. In order to increase funds for small states, 
Perkins IV changed the funding formula to direct one-third of any federal appropriations 
that exceed the FY 2006 funding level to small states until they reach the small state mini- 
mum; the remaining two-thirds would be allocated to all states by formula. Since appropria- 


tions have declined since FY 2006, this provision has not yet come into effect.'4 


Federal funds to outlying areas are allotted on a nonformula basis subject to terms specified 
in Sec. 115 of the legislation. In the first year of Perkins IV (FY 2007), Guam received 
$660,000; American Samoa and the Commonwealth of the Northern Mariana Islands each 
received $350,000; and the Republic of Palau received $160,000. !5 Subsequent amounts were 
adjusted based on the amount appropriated and then distributed in equal proportions to 


Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands.'° 


"The U.S. Virgin Islands has consolidated funding status and receives all of its federal education 
funding, including for Perkins and other education programs such as special education, as a single allo- 
cation from the Office of Elementary and Secondary Education. While the U.S. Virgin Islands does 
report CTE performance data to OVAE, they do not report the amounts allocated to Perkins funding 
categories and are therefore excluded from the state-level analyses in this chapter. 

'2 Sec. 111(a). 

'3 Sec. 111(a)(3). 

4 Sec. 111(a)(4). 

'S Perkins IV made the Republic of Palau ineligible for further funding if it enters into an agreement to 
extend U.S. education assistance under the Compact of Free Association. 

'6 Palau is excluded from subsequent year allocations (Sec. 115(d)). Outlying areas, with the exception 
of Puerto Rico and, where noted, the U.S. Virgin Islands, are excluded from the analysis in this chap- 
ter because funds for outlying areas are not subject to the same fiscal requirements as those for other 
state grantees (Sec. 115(b)(2)). 
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Trends in Grant Amounts 


State grant amounts changed between FY 1992, FY 2001, and FY 2009, years that corre- 
spond to the first program year of Perkins II and the second program years of Perkins HI and 
Perkins IV (Exhibit 2.3).!” Because the distribution formula remained essentially the same 
across Perkins III and IV, changes in states’ eligibility from FY 2001 to FY 2009 are a func- 
tion of annual changes in federal appropriations, changes in states’ population and/or three- 
year average per capita income relative to those of other states, and/or the application of the 


hold-harmless provision and other limits on grant size. 


In nominal dollars, total Title I state grants increased by 4 percent between FY 2001 and FY 
2009. As a consequence, grants for the majority of recipients increased. Of the 34 states (ex- 
cluding outlying areas) gaining funds, amounts allocated to five states increased by more than 
15 percent. All five of these states were among the 10 states that experienced the highest 
rates of population growth between 2000 and 2010, and this growth likely contributed to the 


funding increase. !® 


Although most states experienced a nominal increase in their federal grant, not all benefitted 
equally from appropriations added across the two most recent reauthorizations, and in 11 
states resources declined. Again, these changes may result from relative shifts in states’ popu- 
lation, the three-year average of per capita income relative to that of other states, and/or the 


application of the hold-harmless provision and other limits on grant amounts. 


Grants to three outlying areas increased between Perkins III and IV, with Guam gaining 
more than 30 percent and American Samoa and the Commonwealth of the Northern Maria- 
na Islands gaining more than 80 percent. Although grants for these areas were lower under 
Perkins II, these outlying areas also received Perkins funding through Pacific Resources for 
Education and Learning (PREL) (not shown in Exhibit 2.3). Together, these two sources te- 
sulted in total grants to each of these territories that are similar to the funds that they re- 
ceived under Perkins IV, making the proportion of Perkins funds received by these areas 


similar across the different authorizations of the legislation. !9 


'7 These dates were selected to provide roughly equivalent time periods across Perkins I, IT, III, and 
IV, as well as to provide the most recent data available for analysis. 

18U.S. Census Bureau, 2010 Census Data, Population Change By State: 2000-2010 
http://2010.census.gov/2010census/data/ (accessed April 19, 2011). 

1 Personal communication with the Budget Service, Office of Planning, Evaluation and Policy De- 
velopment, Aug. 3, 2010. 
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Exhibit 2.3. 
Perkins Title I state grant amounts, fiscal years 1992, 2001, and 2009, and percent change, by 
state: FY 2001 through FY 2009 


Perkins II Perkins IIT Perkins IV Percent change from 
FY 2001 through 

State FY 1992 FY 2001 FY 2009 FY 2009 
Alabama $18,493,908 $20,036,322 $19,217,606 -4 
Alaska 4,214,921 4,214,921 4,214,921 0 
Arizona 4,813,300 20,178,519 25,047,298 24 
Arkansas 0,276,155 11,925,341 11,989,737 1 
California 95,689,053 120,745,507 128,360,005 6 
Colorado 1,448,761 14,415,073 15,782,973 10 
Connecticut 9,005,327 8,826,329 10,020,303 14 
Delaware 4,214,921 4,468,631 4,803,968 8 
District of Columbia 4,214,921 4,214,921 4,214,921 0 
Florida 41,552,691 51,525,165 60,428,537 17 
Georgia 26,758,908 31,493,636 38,592,850 23 
Hawaii 4,699,626 5,376,800 5,709,941 6 
Idaho 4,699,626 6,619,244 6,499,494 2 
Illinois 37,481,798 41,157,929 44,837,143 

Indiana 22,791,404 24,786,555 25,818,445 4 
Towa 0,662,123 12,381,109 12,103,307 £2 
Kansas 8,940,430 11,370,063 10,961,229 -4 
Kentucky 6,637,536 18,364,632 17,905,647 3 
Louisiana 9,221,631 22,051,050 21,041,943 5 
Maine 4,695,577 5,376,800 5,709,941 6 
Maryland 3,742,757 15,994,426 16,440,022 3 
Massachusetts 7,429,978 17,323,922 18,687,903 8 
Michigan 34,720,846 38,255,683 40,835,345 7 
Minnesota 5,092,540 17,410,608 17,697,927 2 
Mississippi 2,364,726 13,920,402 13,363,550 -4 
Missouri 9,059,451 22,506,237 23,405,180 4 
Montana 4,214,921 5,268,996 5,363,650 

Nebraska 5,917,914 7,138,876 7,053,557 -1 
Nevada 4,699,626 5,854,216 8,031,665 37 
New Hampshire 4,699,626 5,376,800 5,709,941 6 
New Jersey 21,151,258 22,257,214 24,078,336 

New Mexico 6,595,354 8,559,863 8,858,892 4 
New York 52,699,128 52,486,933 57,403,836 

North Carolina 28,486,370 29,975,525 35,752,471 19 
North Dakota 4,214,921 4,214,921 4,214,921 0 
Ohio 41,619,711 44,682,695 45,028,414 1 
Oklahoma 13,368,617 16,119,667 15,094,180 -6 


See notes at end of table. 
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Exhibit 2.3.— cont. 
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Perkins Title I state grant amounts, fiscal years 1992, 2001, and 2009, and percent change, by 


State 

Oregon 
Pennsylvania 
Rhode Island 
South Carolina 
South Dakota 
Tennessee 
Texas 

Utah 
Vermont 
Virginia 
Washington 
West Virginia 
Wisconsin 


Wyoming 


American Samoa 


Guam 


Northern Mariana Islands 


Puerto Rico 


Virgin Islands 


Freely associated states 


Indian set-aside 


Other (non-State allocations) 


Total 


state: FY 2001 through FY 2009 


Perkins IT 


FY 1992 
$10,379,115 
41,635,031 
4,699,626 
16,293,814 
4,214,921 
20,831,856 
71,509,430 
8,372,087 
4,214,921 
21,516,428 
16,653,997 
8,009,762 
18,463,176 
4,214,921 


191,336 
503,513 
191,336 
17,816,604 
509,173 

0 
12,259,166 
898,256 
953,968,830 


Perkins IIT 


FY 2001 
$13,191,901 
42,540,576 
5,376,800 
17,647,448 
4,328,867 
22,531,516 
86,234,261 
12,453,906 
4,214,921 
24,827,445 
21,232,147 
8,428,617 
21,603,995 
4,214,921 


190,000 
500,000 
190,000 
19,089,614 
567,534 

0 

13,750,000 
10,010,000 
1,100,000,000 


Perkins IV 


FY 2009 
$14,063,250 
44,795,856 
5,709,941 
19,078,798 
4,294,134 
23,882,364 
92,532,081 
12,925,301 
4,214,921 
25,292,041 
21,617,410 
8,428,617 
21,594,496 
4,214,921 


347,509 
655,304 
347,509 
18,458,484 
605,536 
158,862 
14,511,388 
2,902,278 
1,160,911,000 


Percent change from 
FY 2001 through 
FY 2009 


7 
5 


Sc 


Co OO NNO FN DW 


Exhibit reads: The Title I allocation for Alabama was $20,036,322 in 2001 and $19,217,606 in FY 2009, a decrease of about 4 percent. 


n/a Not applicable. 


SOURCE: U.S. Department of Education, Office of the Deputy Secretary, Budget Service. State Funding History Tables by Program, 


http://www.ed.gov/about/overview/budget/history/index.html?src=rt (FY 2001 data; accessed June 16, 2010); and 


http://www2.ed.gov/about/overview/budget/statetables/11stbyprogram.pdf (FY 2009 data, accessed Februaty 1, 2011). 


In FY 2011, overall Perkins IV funding declined by about 12 percent (in nominal dollars) 
when Conetess eliminated Title IH funding and reduced Title I basic grants by about 4 per- 


cent from the previous year. As a result, grants made to all states declined (Exhibit 1.4). The 


percent change ranged from a low of about -1.9 percent to as much as -19.7 percent from 


FY 2010 to FY 2011, reflecting states’ relative population and income levels as used in the 


formula calculations. 
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Exhibit 2.4. 
Estimated federal grants to states before and after the FY 2011 reduction in Title I funds 
and elimination of Title II and percent difference, by state: FY 2010 through FY 2011 


Amount Percent 
FY 2010 total Estimated FY 2011 difference difference 
(base grant plus total FY 2010 through FY 2010 through 
State Tech Prep) (base grant) FY 2011 FY 2011 
Alabama $21,169,358 $19,175,065 -$1,994,293 -9 
Alaska 4,465,084 4,214,921 -250,163 -6 
Arizona 26,950,635 24,555,169 -2,395,466 -9 
Arkansas 12,905,743 11,403,795 -1,501,948 -12 
California 139,243,327 117,708,010 -21,535,317 -15 
Colorado 17,242,558 15,573,228 -1,669,330 -10 
Connecticut 10,831,245 9,564,397 -1,266,848 -12 
Delaware 5,052,516 4,618,602 -433,914 -9 
District of Columbia 4,349,598 4,214,921 -134,677 -3 
Florida 64,193,572 58,075,608 -6,117,964 -10 
Georgia 41,807,825 38,228,368 -3,579,457 -9 
Hawaii 6,121,451 5,468,243 -653,208 -11 
Idaho 7,006,710 6,429,955 -576,755 -8 
Illinois 49,157,223 41,593,212 -7,564,011 -15 
Indiana 28,052,743 25,109,985 -2,942,758 -10 
lowa 13,208,250 11,963,946 -1,244,304 -9 
Kansas 11,721,389 10,245,408 -1,475,981 -13 
Kentucky 19,767,916 17,905,647 -1,862,269 -9 
Louisiana 23,230,400 21,041,943 -2,188,457 -9 
Maine 6,235,453 5,468,243 -767,210 -12 
Maryland 18,424,911 15,367,652 -3,057,259 -17 
Massachusetts 20,565,053 17,912,559 -2,652,494 -13 
Michigan 44,594,721 39,142,926 -5,451,795 -12 
Minnesota 19,395,706 16,754,034 -2,641,672 -14 
Mississippi 14,753,419 13,363,550 -1,389,869 -9 
Missouti 24,871,527 22,165,679 -2,705,848 -11 
Montana 5,825,871 5,170,438 -655,433 -11 
Nebraska 7,525,881 6,816,893 -708,988 -9 
Nevada 8,609,174 8,441,770 -167,404 -2 
New Hampshire 6,086,100 5,468,243 -617,857 -10 
New Jetsey 26,071,027 22,219,837 -3,851,190 -15 
New Mexico 9,279,588 8,028,287 -1,251,301 -13 
New York 66,954,138 53,752,353 -13,201,785 -20 
North Carolina 38,904,958 35,259,396 -3,645,562 -9 


See notes at end of table. 
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Exhibit 2.4.—cont. 
Estimated federal grants to states before and after the FY 2011 reduction in Title I funds 
and elimination of Title II and percent difference, by state: FY 2010 through FY 2011 


Amount Percent 
FY 2010 total Estimated FY 2011 difference difference 
(base grant plus total FY 2010 through FY 2010 through 
State Tech Prep) (base grant) FY 2011 FY 2011 
North Dakota $4,528,072 $4,214,921 -$313,151 -7 
Ohio 49,171,479 42,750,001 -6,421,478 -13 
Oklahoma 16,664,042 15,094,180 -1,569,862 -9 
Oregon 15,361,296 13,566,146 -1,795,150 -12 
Pennsylvania 49,131,116 42,285,519 -6,845,597 -14 
Rhode Island 6,048,484 5,468,243 -580,241 -10 
South Carolina 20,567,894 18,256,242 -2,311,652 -11 
South Dakota 4,697,519 4,214,921 -482,598 -10 
Tennessee 25,366,106 23,208,260 -2,157,846 -9 
Texas 101,081,082 88,684,893 -12,396,189 -12 
Utah 14,197,655 12,544,692 -1,652,963 -12 
Vermont 4,452,109 4,214,921 -237,188 -5 
Virginia 27,720,438 23,932,351 -3,788,087 -14 
Washington 23,061,524 20,335,265 -2,726,259 -12 
West Virginia 9,305,231 8,428,617 -876,614 9 
Wisconsin 23,499,683 20,782,582 -2,717,101 -12 
Wyoming 4,448,650 4,214,921 -233,729 -5 
Puerto Rico 20,378,250 18,458,484 -1,919,766 -9 
Virgin Islands 655,450 571,147 -84,303 -13 


Exhibit reads: The total Perkins state grant amount for Alabama was $21,169,358 in FY 2010 and $19,175,065 in FY 2011, a difference of - 
$1,994,293 or about -9 percent. 

SOURCE: U.S. Department of Education, Office of the Deputy Secretary, Budget Service, Fiscal Year 2010-FY 2012 President's Budget 
State Tables for the U.S. Department of Education http://www2.ed.gov/about/ovetrview/budget/statetables/index.html; and U.S. 
Department of Education, Office of Vocational and Adult Education. (2011). Program Memorandum: Revised Estimated Federal Fiscal 
Year (FY) 2011 State Allocations under the Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV) 


State Categorical Funding for CTE 


CTE instruction is often costlier to provide than academic instruction. These greater costs 
are mainly due to the smaller classes associated with CTE instruction: schools adopt lower 
student-teacher ratios for CTE because of the high cost of specialized equipment and the 
potentially higher risk associated with its use (Klein 2001). To offset these added expenses, 
some states have adopted categorical funding for CTE to steer supplemental funding to local 
providers. In many cases, these state resources ate supplemented by local resources and ex- 


ceed the contribution made by federal Perkins funds to local programs. 
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Findings from the survey of state directors indicate that about half of states provided some 
form of categorical aid to support secondary CTE programs.” Of the 48 secondary state di- 
rectors who provided information on categorical funding, 22 reported that their states pro- 
vided some form of dedicated funding in FY 2008 for CTE in secondary schools. This result 
is not consistent with the findings in a previous report, which indicated that in FY 2001, 40 
states reported offering some form of categorical funding for CTE services in secondary 
schools (Klein 2001).21 Among the 22 state directors who reported categorical statewide 
support for CTE, 16 indicated that the source of the funds was the state, seven reported lo- 
cal sources, and four federal sources. At the postsecondary level, state directors reported that 
CTE categorical funding was available during the 2008-09 program year in roughly two- 
fifths (18) of the 42 states responding to the question. With regard to the source of funds, 
the majority of directors (13) indicated they did not know the funding source.”# 


State Uses of Perkins Funds 


A small portion of Perkins IV funds allotted to states may be harnessed by state staff for 
grant administration, leadership, and other legislated uses. The following section details the 
statutory requirements associated with these funds and how states are using them to carry 


out the legislation’s provisions. 


Eligible Agencies 


Perkins IV requires states to designate or create a single board—called an eligible agency—to be 
responsible for the administration or supervision of CTE in the state.?> Generally, this board 
is located within the state department of education, with 40 of the 55 total Perkins state 
grantees charging their K—12 education agency with Perkins IV oversight. Nine states chose a 
postsecondary system office or institution to serve in this role, and six states designated a 
state department of CTE or workforce development agency.** The choice of eligible agency 
likely reflects the relative amounts of funds flowing into each education level, state adminis- 


trative capacities, historical precedent, and other state-specific factors. 


20 Secondary and Postsecondary State Director Surveys, 2009. 

21 The results may reflect changes in state funding formulas over time, directors’ misunderstanding of 
the survey question, or different interpretations of the question’s intent. For example, directors in 
states providing categorical funding for stand-alone area CTE centers might have responded negative- 
ly if they believed that the question only applied to comprehensive high schools. 

2 The response rate for some question items was less than 85 percent; results should be interpreted 
with caution. 

3 Sec. 3(10). 

4 This total includes territories and outlying areas. http://www.careertech.org/state_profile/ (ac- 
cessed June 10, 2010). 
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The eligible agency is directly responsible for developing, submitting, and implementing the 

Perkins IV state plan and evaluating all Title I-related programs, services, and activities.*° The 
eligible agency also is responsible for administering funds in accordance with Perkins IV’ stat- 
utory provisions. States are required to divide their Title I state grant into the following cate- 


gories: 


e State Administration: up to 5 percent or $250,000 (whichever is greater) for use 


in administering the state plan. 


e State Leadership: up to 10 percent for use in carrying out state programs, with 
not more than 1 percent for individuals in state institutions and not less than 
$60,000 or more than $150,000 for services to prepare individuals for nontradi- 
tional fields. 


e Grants to Local Providers: at least 85 percent for secondary and postsecondary 


education programs. 


State Administration 


Congress allows states to use up to 5 percent or $250,000 of their Title I grant (whichever is 
greater) for administering Perkins IV. Allowable uses include (1) developing the state plan, 
(2) reviewing local plans, (3) monitoring and evaluating program effectiveness, (4) ensuring 
compliance with federal law, (5) providing technical assistance, and (6) supporting and de- 


veloping state data systems relevant to Perkins provisions.” 


Just under half of states made full use of the state administration provision to offset the 
staffing and resource costs associated with Perkins IV administration. Twenty-four of the 51 
states for which FY 2010 data are available reserved the maximum amount available for state 


administrative activities (Exhibit 2.5). 


Federal law stipulates that states provide a dollar-for-dollar match for funds used for state 
administrative purposes, although states are not required to obligate these funds for any spe- 
cific purpose.?’ It appears that 19 of 50 states for which data are available are contributing 
state funds in excess of the federal match requirement. While in some instances, additional 
funds are simply due to states’ rounding of their contribution, in others, the state contribu- 


tion substantially exceeds the match requirement. 


25 Sec. 121. 
°6 Sec. 112(a)(3). 
27 Sec. 112(b). 
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Exhibit 2.5. 
Amount and percentage of state Perkins Title I and II funds allocated to administration 
and amount and percentage of state match, by state: FY 2010 


Percent of 
Total Title I State Percent of total total state 
and II administration state Perkins Matching state Perkins 

State funds amount allocation contribution allocation 
Alabama* $21,169,358 $1,028,468 5 $1,028,468 5 
Alaska 4,465,084 250,000 6 250,000 6 
Arizona 26,950,635 1,253,518 5 2,358,900 9 
Arkansas* 12,905,743 645,287 5 1,000,000 8 
California 139,243,327 6,399,575 5 6,399,575 5 
Colorado* 17,242,558 792,447 5 792,447 5 
Connecticut* 10,831,245 541,562 5 51,189,507 473 
Delaware 5,052,516 250,000 5 71,000,000 1405 
District of Columbia* 4,349,598 250,000 6 250,000 6 
Florida* 64,193,572 1,350,000 2 1,350,000 2 
Georgia* 41,807,825 2,090,390 5 2,090,390 5 
Hawaii* 6,121,451 306,073 5 523,137 9 
Idaho* 7,006,710 350,335 5 n/a n/a 
Illinois 49,157,223 2,255,395 5 2,255,395 5 
Indiana 28,052,743 494,923 2 494,923 2 
lowa 13,208,250 $598,197 5 598,197 5 
Kansas* 11,721,389 586,069 5 586,069 5 
Kentucky* 19,767,916 988,396 5 2,167,818 11 
Louisiana* 23,230,400 1,161,520 5 1,161,520 5 
Maine* 6,235,453 311,773 5 311,773 5 
Maryland* 18,424,911 921,246 5 921,246 5 
Massachusetts 20,565,053 600,000 3 600,000 3 
Michigan 44,594,721 2,047,648 5 2,047,648 5 
Minnesota* 19,395,706 969,785 5 969,785 5 
Mississippi 14,753,419 668,178 5 668,178 5 
Missouti 24,871,527 1,134,685 5 1,667,627 7 
Montana 5,825,871 269,892 5 285,091 5 
Nebraska* 7,525,881 376,294 5 376,294 a 
Nevada* 8,609,174 430,458 5 488,233 6 
New Hampshire 6,086,100 304,305 5 285,498 5 
New Jetsey* 26,071,027 1,303,551 5 1,303,551 5 
New Mexico 9,279,588 422,287 5 422,287 5 
New York 66,954,138 1,400,000 2 1,400,000 2 
North Carolina 38,904,958 1,795,580 5 1,800,000 5 
North Dakota* 4,528,072 250,000 6 2,100,000 46 


See notes at end of table. 
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Exhibit 2.5.—cont. 
Amount and percentage of state Perkins Title I and II funds allocated to administration 
and amount and percentage of state match, by state: FY 2010 


Percent of 
Total Title I State Percent of total total state 
and IT administration state Perkins Matching state Perkins 

State funds amount allocation contribution allocation 
Ohio $49,171,479 $2,233,195 5 $2,233,195 5 
Oklahoma 16,664,042 754,709 5 754,709 5 
Oregon* 15,361,296 768,064 5 768,064 a 
Pennsylvania 49,131,116 2,244,788 5 2,244,788 5 
Rhode Island* 6,048,484 285,497 5 437,252 re 
South Carolina* 20,567,894 956,407 5 956,407 5 
South Dakota 4,697,519 250,000 5 561,852 12 
Tennessee* 25,366,106 1,268,305 5 ,268,305 5 
Texas 101,081,082 3,290,344 3 3,290,344 3 
Utah* 14,197,655 709,882 5 1,539,536 11 
Vermont* 4,452,109 250,000 6 282,668 6 
Virginia 27,720,438 1,265,133 5 ,669,645 6 
Washington 23,061,524 1,051,234 5 1,051,234 5 
West Virginia 9,305,231 421,431 5 1,203,000 13 
Wisconsin 23,499,683 1,069,723 5 119,915,408 510 
Wyoming* 4,448,650 250,000 6 442,593 10 
Puerto Rico* 20,378,250 n/a n/a n/a n/a 
Total 1,244,255,700 51,866,549 4 299,762,557 24 


Exhibit reads: In FY 2010, Alabama allocated $1,028,468, or 5 percent, of its Perkins funds for administration. Alabama’s matching state 
contribution was also $1,028,468, which is equivalent to 5 percent of the state’s Perkins funds. 

n/a Not applicable 

* State merged (or in the case of Alabama, partially merged) Title II funds with Title I funds in program year 2009-10 or earlier. 
SOURCE: U.S. Department of Education, Office of Vocational and Adult Education, 2011, Internal Perkins Data Base System and State 
Consolidated Annual Report data system information for FY 2010. 


It is unknown whether these funds represent additional resources that would not otherwise 
be provided for CTE, or simply reflect funds already legislated by state formula. This study 
did not collect comprehensive data on the source of state matching funds, but it appears that 
some states may be classifying state categorical funding for CTE as meeting the state match 
requirement. For example, in the survey of state directors, one director explained that the 
matching funds reported in excess of the required amount represent all the funds spent at 
the state level for the administration of CTE programs. State directors in three states report- 
ed that the matching funds come from the state general fund or general revenue for the state 


agency’s administrative functions. 
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The variation in matching amounts raises a question, however, about the intent of the state 
match requirement. Some states may be providing supplemental funding for CTE admin- 
istration and programs to meet the match requirement, whereas others may be recording tre- 
soutces that would be committed to CTE support in the absence of the match requirement. 


The legislation does not specify how matching funds should be obligated. 


State Leadership 


States may reserve up to 10 percent of their Title I funds as state leadership funds, which can 
be used to carry out a set of required and permissible activities.2> In FY 2010, forty-two 
states allocated 10 percent for these activities, nine states at least 8 percent but less than 10 
percent, and one state 5 percent.” There are nine required activities that states must imple- 
ment using federal funds, such as assessing CTE programs funded under Perkins IV and 
providing technical assistance to eligible recipients. Permissible uses cover 17 activities, rang- 
ing from establishing articulation agreements between secondary and postsecondary CTE 


programs to supporting CTE student organizations. 


Perkins IV expands upon the required and permissible state leadership activities in Perkins LI 
(Exhibit 2.6). For example, among required activities, Perkins IV includes providing guidance 
on the content of professional development programs, such as “the effective use of scientifi- 
cally based research and data to improve instruction.”’*° Congress also added a requirement 
that states provide technical assistance for eligible recipients, which was previously a permis- 
sible use. The reauthorization added seven new permissible activities, including entrepre- 
neurship education and training and the development of technical skill assessments, among 


others. 


8 See Sec. 124(b) for a list of required uses and Sec. 124(c) for a list of permissible uses of funds. 

°° Internal Perkins Database System and State Consolidated Annual Report data system information 
for FY 2010. 

30 Sec. 124(b)(3). 
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Exhibit 2.6. 
Summary of required and permissible state leadership activities: 
Perkins II and IV 


Requited uses of state leadership funds Perkins II _ Perkins IV 
Preparing individuals for nontraditional fields* Vv Vv 
Assessing CTE programs* 

Developing, improving, or expanding the use of technology in CTE* 

Professional development programs* 

Integration of academics with vocational and technical education* 

Supporting partnerships among secondary, postsecondary, adult education, and other local institutions* 


Serving individuals in state institutions 


<_ < < < < < €< 


Programs for special populations* 


e<<e4e4eeee 


Technical assistance for eligible recipients" 


Permissible uses of state leadership funds Perkins II Perkins IV 
Technical assistance for eligible recipients” v 


Career guidance and academic counseling* 


< 


Establishing articulation agreements* 

Cooperative education? 

Vocational student organizations (VSOs) 

Public charter schools operating secondary CTE programs 
Programs offering exposure to all aspects of an industry 
Family and consumer sciences programs 

Education and business partnerships* 

Improving or developing new CTE courses and initiatives* 


CTE programs for adults and school dropouts 


<_ < < < < < < < fs Ff F< 


Job and continuing education support for CTE students 

Initiatives to facilitate transition of subbaccalaureate CTE students into baccalaureate degree programs 
Incentive grants to local grant recipients 

Entrepreneurship education and training 

Developing technical skills assessments 


Developing and enhancing data systems to collect academic and employment outcome data 


<_ < < < < < < < fs fs fs fs Sf Ss 


CTE faculty and other personnel recruitment and retention 


Occupational and employment information resources Vv 


Exhibit reads: Required and permissible uses of state leadership funds increased by one required and seven permissible activities between 
Perkins III and Perkins IV. 

* Focus of activity was the same in Perkins IIT and IV, but language and/or section details changed. 

“Changed from a permissible to a required use from Perkins III to IV. 

> Combined with support for education and business partnerships in Perkins IV (Perkins IV, Sec. 124(c)(8). 

NOTE: For full description of activities, see Perkins IV, Sec. 124(b) and Sec. 124(c). 

SOURCE: Perkins IV’, Sec. 124(b) and Sec. 124(c). 
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Secondary and postsecondary state directors were asked to report on the proportion of lead- 
ership funds that were dedicated to different required leadership activities in the 2008-09 
program year (Exhibit 2.7). According to the state directors, the largest percentage of funds 
(about 20 percent) went to technical assistance for eligible grant recipients, followed by 
providing professional development activities (16 and 15 percent at the secondary and post- 
secondary levels, respectively) and assessing CTE programs (11 and 10 percent at the sec- 
ondary and postsecondary levels, respectively). Reflecting the limit the legislation places on 
funds for this purpose, directors reported the lowest percentages for serving individuals in 
state institutions, such as state correctional facilities and institutions serving disabled students 


(about 5 percent).*! 


Exhibit 2.7. 
Average percentage of state leadership funds allocated for required activities: 
Program year 2008-09 


Activity Secondary Postsecondary 
Technical assistance for eligible recipients 20 21 
Providing professional development 16 15 
Assessing CTE programs 11 10 
Strengthening the integration of academic and CTE instruction 9 8 
Expanding the use of technology in CTE programs 8 7 
Preparing individuals for nontraditional employment f 7 


Supporting partnerships among secondary, postsecondary, adult education, 
and other local institutions 8 9 


Support for programs for special populations that lead to high-skill, high-wage, 


high-demand occupations* 8 6 
Serving individuals in state institutions, such as state correctional and 

institutions serving disabled students 6 5 

Number of responses 34 22 

Don’t know/ no response 16 26 


Exhibit reads: States reported spending an average of 20 percent at the secondary level and 21 percent at the postsecondary level of state 
leadership funds on technical assistance for eligible recipients. 

N=50 (secondary); 48 (postsecondary). 

“Reflects the language included in the Act (Section 124(b)(8)). The category included in the survey was “Supporting programs that prepare 
special education students for entry into high-skill, high-wage, high-demand occupations.” 

NOTE: Responses wete not required to sum to 100 percent. 

SOURCE: Secondary and Postsecondaty State Director Surveys, 2009. 


31 Comparisons of trend data across the 1998 and 2006 legislation are not possible due to differences 
in survey questions included in the 2004 NAVE national assessment; the 2004 study combined a sub- 
set of required and permissible uses of state leadership funds, while the 2010 survey focused solely on 
required uses. 
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Perkins IV introduced a new requirement that local providers offer one or more POS to de- 
liver CTE services. POS are coherent programs of CTE instruction that (1) incorporate sec- 
ondary and postsecondary education elements; (2) offer rigorous technical content aligned 
with challenging academic standards in a coordinated, nonduplicative progression of courses 
aligning secondary and postsecondary education; (3) may offer high school students the op- 
portunity to participate in dual or concurrent enrollment programs to earn postsecondary 
credits; and (4) lead to an industry-recognized credential or certificate at the secondary level 


ot an associate’s or bachelor’s degree.” 


Some 24 state directors at the secondary level and 19 at the postsecondary level reported di- 
recting state leadership funds to establish POS. At the secondary level, states allocated 
roughly 17 percent of their leadership funds to POS development and 15 percent to imple- 
mentation. Investment was somewhat lower for the postsecondary level: states used about 9 
percent for POS development and 7 percent for POS implementation.*> These data support 
findings on POS discussed in Chapter 4, which indicate that POS development and imple- 


mentation activities tend to be focused mainly at the secondary education level. 


32 Sec. 122()(1). 

33 Secondary and Postsecondary State Director Surveys, 2009. N=50 (secondary); 48 (postsecondary). 
The numbers of state directors who responded that they didn’t know or did not respond were 27 and 
29 at the secondary and postsecondary levels, respectively. Because items on these questions had re- 
sponse rates of less than 85 percent, data should be interpreted with caution. 


31 
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Nontraditional employment 
Perkins IV mandates that states reserve no less than $60,000 or more than $150,000 of their 


state leadership funds for services that prepare individuals for nontraditional fields.*4 


State support for nontraditional services declined from Perkins HI to IV in nominal dollars. 
On average, states expended $91,500 in FY 2006 versus $80,983 in FY 2010 (Exhibit 2.8).°5 
The number of states spending at the minimum of $60,000 increased from 25 in FY 2006 to 
32 in FY 2010. Two fewer states in FY 2010 than in FY 2006 spent the maximum amount; 
11 states directed $150,000 to nontraditional services in FY 2006 and 9 in FY 2010. 


Exhibit 2.8. 


Number of states at different funding levels for services that prepare individuals for 


nontraditional fields and the average amount allocated: 


FY 2006 and FY 2010 
Nontraditional funding level FY 2006 FY 2010 
Minimum amount ($60,000) 25 32 
More than $60,000 but less than $150,000 16 10 
Maximum amount ($150,000) 11 9 
Average allocation $91,500 $80,983 


Exhibit reads: In FY 2006, 25 states allocated $60,000 for services that prepare individuals for nontraditional fields. The average amount 


that states allocated for this purpose was $91,500. 
N=52 (FY 2006); 51 (FY 2010). 


SOURCE: U.S. Department of Education, Office of Vocational and Adult Education, 2011, Internal Perkins Data Base System and State 


Consolidated Annual Report data system information for FY 2010. 


Many states have staff to coordinate state gender equity and activities for special populations 
at the secondary and postsecondary levels, even though the requirement that states do so 
was eliminated in Perkins II. More than one-half (29 of 44) of states responding at the sec- 
ondary level and 23 of 36 states at the postsecondary level reported that they had a gender 
equity or special populations coordinator, or both, on staff in the 2000-01 program year 
(Exhibit 2.9) (Silverberg et al. 2004). 


34 Sec. 112(1)(2)(B). “Nontraditional fields” are defined in Sec. 3(17) as “occupations or fields of work, 
including careers in computer science, technology, and other emerging high-skill occupations, for 
which individuals from one gender comprise less than 25 percent of the individuals employed in each 
such occupation or field of work.” 

55 State-level allocation comparisons are between FY 2006 and FY 2009. FY 2006 is the first year for 
which reliable and complete data are available, and corresponds to the second to last year of Perkins 
III. FY 2009 corresponds to the second year of Perkins IV, and is the most recent year for which data 
were available at the time of writing. 
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Exhibit 2.9. 
Number of states reporting gender equity and special populations coordinators at the 
secondary and postsecondary levels: Program years 2000—01 and 2008-09 


2000-01" 2008-09” 
Position Secondary Postsecondary Secondary Postsecondaty 
Gender equity coordinator only 10 6 6 6 
Special populations coordinator only 6 5 4 3D 
Both gender and special populations coordinator 13 12 23 13 
Neither 15 13 16 19 
No response 2 10 il 5 
Average numbet of FTEs dedicated to these positions‘ 1,5 1.5 1.1 1.0 


Exhibit reads: In program year 2000-01, 10 states reported having a gender equity coordinator only at the secondary level. The average 
number of full-time equivalent positions (FTEs) dedicated to these positions was 1.5 at the secondary level in 2000-01. 


“N=46. 
>N=50 (secondary); 48 (postsecondary). 


“ Among states reporting one or both of these positions in full-time equivents (FTEs). 
NOTE: The results may overestimate the number of positions because coordinators shared across levels may have been reported for both 


levels in some states. 
SOURCE: Silverberg et al. (2004); White et al. (2004); and Secondary and Postsecondary State Director Surveys, 2009. 


Some 33 of 49 states at the secondary level and 24 of 43 states at the postsecondary level re- 
ported having a gender equity or special populations coordinator, or both, in the 2008-09 
program year. The number of full-time equivalent positions dedicated to these positions de- 
clined between 2000-01 and 2008-09. In states that reported having one or both of these 
positions in 2000-01, the positions represented an average of 1.5 FTEs at the secondary and 
postsecondary levels. In 2008-09, the average FTEs fell to 1.1 and 1.0 at the secondary and 
postsecondary education levels, respectively, indicating that more of these positions were 


part time than in 2000-01. 
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Individuals in state institutions 
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States are required to use a portion of their state leadership funds to serve individuals in state 


institutions, which include correctional facilities and facilities serving individuals with disabil- 


ities.5° Although the legislation does not specify a minimum amount, it caps spending at no 


more than 1 percent of a state’s Title I funds. 


In both FY 2006 and FY 2010, states spent, on average, about 0.8 percent of their Title I 


funds on programs serving individuals in state institutions (Exhibit 2.10). The majority of 


states allocated more than 0.8 percent of Title I funds for this purpose in both years (34 in 


FY 2006 and 32 in FY 2010). 


Exhibit 2.10. 


Average and total amounts, and the average percentage of Title I funds allocated to serve 


individuals in state institutions, and the distribution of states by the percentage of Title I 


funds allocated for this purpose: FY 2006 and FY 2010 


FY 2006 
Average amount allocated $181,020 
Total amount allocated” 9,413,014 
Average percentage of Title I funds allocated* 1 
Number of states allocating: 
Less than 0.3 percent 6 
0.3 to 0.8 percent 12 
More than 0.8 percent 34 


FY 2010 
$179,790 
9,169,303 


9 
10 
32 


Exhibit reads: In FY 2006, states allocated an average amount of $181,020 for programs to serve individuals in state institutions, and 34 


states allocated more than 0.8 percent of Title I funds for this purpose. 
N=52 (FY 2006); 51 (FY 2010). 


“The national average is the average percentage of Title I funds dedicated for this purpose across all states. 


b . ; 
The total amount allocated is the sum of allocations across all states. 


SOURCE: U.S. Department of Education, Office of Vocational and Adult Education, 2011, Internal Perkins Data Base System and State 
Consolidated Annual Report data system information for FY 2010. 


36 Sec. 112(a)(2). 
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Reserve funds 
Perkins III increased the minimum amount of resoutces that states must distribute to local 
programs from 75 percent to 85 percent of the overall state grant. At the same time, Con- 
gress offered states the option of creating a reserve fund of up to 10 percent of local sub- 
grantee funding (or 8.5 percent of the state’s total grant), which gave states flexibility in 


adjusting to the new requirement.*’ 


Reserve funds are allocated to eligible recipients through means other than the statutory 
formula chosen by the state, such as alternative formulas or competitions. Perkins ILI re- 
quired states to target their reserves to local programs that (1) were located in rural areas, (2) 
had high numbers of vocational students, (3) had large percentages of vocational students, or 
(4) had experienced funding losses due to within-state formula changes introduced in the 
1998 legislation. Because Perkins IV introduced no changes to the within-state allocation 
formula, the latter provision was dropped in the 2006 legislation, along with a requirement 


that states spend funds on at least two of the specified categories.** 


A growing number of states are making use of the reserve provision (Exhibit 2.11); between 
FY 2006 and FY 2010 the number of states using the reserve fund increased from 24 to 41. 
As a result, the total amount of reserve funds recorded across states grew from roughly 
$34.4 to $51.7 million, or about 50 percent. Among states opting for the reserve fund, the 
average percentage of local funds allocated was 7.6 percent in FY 2006 and 7.7 percent in 
FY 2010. Ten states chose the full 10 percent permitted in FY 2006, and 21 did so in FY 
2010. 


States have discretion over how to divide reserve funds between the secondary and postsec- 
ondary education levels (Exhibit 2.12). As with formula-allocated Title I funds, in FY 2010 
the bulk of reserve funds across all states was allocated to the secondary level (63 percent). 
Fourteen states with a reserve fund used the funds exclusively for secondary education. By 


contrast, just five states allocated their entire reserve fund to the postsecondary level. 


37 Sec. 112(a)(1). 
38 Sec. 112(c). 
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Exhibit 2.11. 
Percentage and amount of Title I local funds used as reserve funds, by state: FY 2006 and 
FY 2010 
FY 2006 FY 2010 
Reserve Reserve 
(percentage of Reserve (percentage of Reserve 

State local funds) amount local funds) amount 
Avetage percentage and 

total reserve amount* 8 $34,379,591 8 $51,728,212 
Alabama 7 1,121,513 7 1,223,877 
Alaska 10 358,268 10 358,268 
Arizona 9 1,763,956 4 818,590 
Arkansas * * 600,000 
California * * 7 . 
Colorado 10 1,329,388 10 1,486,522 
Connecticut 4 344,613 8 736,526 
Delaware * * 10 409,068 
District of Columbia " = = = 
Florida 7 3,818,842 5 2,872,902 
Georgia 5 1,554,931 10 3,553,664 
Hawaii * = 2 78,049 
Idaho = = 7 410,915 
Illinois 3 * * * 
Indiana 6 1,451,510 = - 
Iowa ss 5 2 150,000 
Kansas * * 10 996,318 
Kentucky ms * 8 1,344,218 
Louisiana * + 10 1,974,584 
Maine 10 491,258 10 530,014 
Maryland * = 717,585 
Massachusetts a 428,350 2 300,000 
Michigan = . 1 300,000 
Minnesota * * 10 1,651,822 
Mississippi - * * - 
Missouri 10 2,020,867 10 1,928,963 
Montana 10 463,856 10 458,817 
Nebraska 500,000 8 500,000 
Nevada 2 156,892 10 695,190 


See notes at end of table. 
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Percentage and amount of Title I local funds used as reserve funds, by state: FY 2006 and 
FY 2010 


State 

New Hampshire 
New Jersey 
New Mexico 
New York 
North Carolina 
North Dakota 
Ohio 

Oklahoma 


Oregon 
Pennsylvania 
Rhode Island 
South Carolina 
South Dakota 
Tennessee 
Texas 

Utah 
Vermont 
Virginia 
Washington 
West Virginia 
Wisconsin 


Wyoming 


Puerto Rico 


Exhibit 2.11.—cont. 


FY 2006 


Reserve 
(percentage of 
local funds) 


i) 
10 
10 


10 
10 


* 


Reserve 
amount 


$245,629 
2,100,839 
787,405 


* 
* 


* 


3,873,461 
1,355,174 

n/a 
1,936,992 


* 
* 


368,831 


x 
4,943,999 
* 
* 
* 


1,077,164 


* 


1,885,853 


* 


* 


FY 2010 


Reserve 
(petcentage of 
local funds) 


5 
8 
10 


* 


Reserve 
amount 


$258,659 
1,662,028 
717887 
Z 
2,034,991 
382,775 
3,801,957 
1,283,005 
1,305,710 
‘ 
519,199 
1,755,469 
366,357 
2,156,119 
8,013,031 
400,000 
187,845 


* 


1,787,097 


* 


1,000,191 


* 


* 
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Exhibit reads: Alabama allocated $1,121,513 or 6.6 percent of local funds to the reserve fund in FY 2006 and $1,223,877 or 7.0 percent in 


FY 2010. 


* State did not have a reserve fund. 


“ Average and total calculated for the 24 states in FY 2006 and the 41 states in FY 2010 that had a reserve fund. 


SOURCE: U.S. Department of Education, Office of Vocational and Adult Education, 2011, Internal Perkins Data Base System and State 


Consolidated Annual Report data system information for FY 2010. 
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Exhibit 2.12. 
Percentage and amount of reserve funds allocated to the secondary and postsecondary 
levels, by state: FY 2010 


Secondary Postsecondary 
Reserve amount Amount Percent Amount Percent 

Total reserve amount $51,728,212 $32,375,776 63 $19,352,437 37 
Alabama 1,223,877 1,223,877 100 0 0 
Alaska 358,268 310,000 87 48,268 14 
Arizona 818,590 818,590 100 0 

Arkansas 600,000 600,000 100 0 

California * gs = * = 
Colorado 1,486,522 891,913 60 594,609 40 
Connecticut 736,526 405,089 55 331,437 45 
Delaware 409,068 409,068 100 0 0 
District of Columbia - * Fe * - 
Florida 2,872,902 2,517,324 88 355,578 12 
Georgia 3,553,664 1,776,832 50 1,776,832 50 
Hawaii 78,049 39,025 50 39,025 50 
Idaho 410,915 0 0 410,915 100 
Illinois * * * * * 
Indiana a 7 = ” 
lowa 150,000 0 0 150,000 100 
Kansas 996,318 498,159 50 498,159 50 
Kentucky 1,344,218 739,320 55 604,898 45 
Louisiana 1,974,584 0 0 1,974,584 100 
Maine 530,014 349,809 66 180,205 34 
Maryland 717,585 466,430 65 251,155 35 
Massachusetts 300,000 300,000 100 0 

Michigan 300,000 300,000 100 0 

Minnesota 1,651,822 693,765 42 958,057 58 
Mississippi - ~ i % ee 
Missouti 1,928,963 1,928,963 100 0 0 
Montana 458,817 229,408 50 229,409 50 
Nebraska 500,000 275,000 55 225,000 45 
Nevada 695,190 195,190 28 500,000 72 
New Hampshire 258,059 258,059 100 0 0 
New Jetsey 1,662,028 1,662,028 100 0 0 
New Mexico 717,887 717,887 100 0 0 
New York * be = ” - 
North Carolina 2,034,991 2,034,991 100 0 0 


See notes at end of table. 
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Exhibit 2.12.—cont. 
Percentage and amount of reserve funds allocated to the secondary and postsecondary 
levels, by state: FY 2010 


Secondary Postsecondary 
Reserve amount Amount Percent Amount Percent 
North Dakota $382,775 $82,775 22 $300,000 78 
Ohio 3,801,957 0 0 3,801,957 100 
Oklahoma 1,283,005 1,283,005 100 0 0 
Oregon 1,305,710 652,855 50 652,855 50 
Pennsylvania of * * * 7 
Rhode Island 519,199 137,068 26 382,131 74 
South Carolina 1,755,469 1,755,469 100 0 0 
South Dakota 366,357 274,768 75 91,589 25 
Tennessee 2,156,119 1,724,895 80 431,224 20 
Texas 8,013,031 5,649,446 71 2,363,585 30 
Utah 400,000 200,000 50 200,000 50 
Vermont 187,845 187,845 100 0 0 
Virginia . ss * - as 
Washington 1,787,097 786,323 44 1,000,774 56 
West Virginia 7 = - = 
Wisconsin 1,000,191 0 0 1,000,191 100 
Wyoming * ok * * * 
Puerto Rico . = i = 7 


Exhibit reads: Alabama allocated $1,223,877 or 100 percent of its reserve fund to the secondary level in FY 2010. 

* State did not have a reserve fund. 

SOURCE: U.S. Department of Education, Office of Vocational and Adult Education, 2011, Internal Perkins Data Base System and State 
Consolidated Annual Report data system information for FY 2010. 


In the 2008-09 program year, state directors reported most often using secondary reserve 

funds to support programs in rural areas (12 respondents), followed by areas with high per- 
centages of CTE students (9 respondents) (Exhibit 2.13). At the postsecondary level, rural 

areas and areas with high numbers of CTE students tied for the most common target cate- 
gories. Case study evidence indicates that states used the reserve fund to encourage innova- 
tion or allocate additional funds to districts and institutions based on need. Although most 
states have established reserve funds, not all have. Administrators from one of the 11 states 
that chose not to establish a reserve fund reported state concerns that a reserve fund might 


be perceived as taking money from districts. 
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Exhibit 2.13. 
Number of states using reserve fund allocation criteria options: 
Program years 2000-01 and 2008-09 


2000-01" 2008-09” 
Criteria Secondary Postsecondary Secondary Postsecondary 
Rutal areas 12 7 12 6 
Areas with high percentages of CTE students 10 6 9 2 
Areas with high numbers of CTE students 7 5 6 
Other® 8 2 4 2 
Don’t know/No response 2 2 16 13 
No resetve/No reserve at level 25 35 17 26 


Exhibit reads: In program year 2000-01, 12 states reported allocating reserve funds at the secondary level to tural areas. 
“N=46. 
>N=50 (secondary); 48 (postsecondary). 


“In 2000-01, “Other” includes “Communities negatively affected by the 1998 Act.” 
NOTE: Respondents could use more than one type of criteria. 
SOURCE: White et al. (2004); Secondary and Postsecondary State Director Surveys, 2009. 


Title | Resources at the Secondary and Postsecondary 
Levels 


Perkins IV provides states with considerable discretion in how they distribute their Title I 
funds across secondary and postsecondary education. This section examines trends in the 
distribution of resources across these education levels and explores whether, and to what ex- 


tent, new legislative features have affected states’ distribution decisions. 


Historically, states have distributed a larger share of Perkins funds to the secondary level, a 
trend that continues with Perkins IV. Nationally, states allocated an average of about 64 per- 
cent of funds to the secondary level in FY 2010, a proportion that has risen by just 2 per- 
centage points since FY 1992 (Exhibit 2.14).°° 


*° These funds exclude the up to 10 percent reserve funding that states may choose to distribute using 
criteria detailed in Sec. 112(c). 
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Exhibit 2.14. 
Percentage of Perkins Title I formula-allocated funds allocated to the secondary and 
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postsecondary levels, fiscal years 1992, 2001, and 2010, and change in the percentage share 
allocated to the secondary level, by state: FY 2001 through FY 2010 


State 


National average” 


Alabama 
Alaska 
Arizona 
Arkansas 
California 
Colorado 
Connecticut 
Delaware 
District of Columbia 
Florida 
Georgia 
Hawaii 
Idaho 
Illinois 
Indiana 
lowa 
Kansas 
Kentucky 
Louisiana 
Maine 
Maryland 
Massachusetts 
Michigan 
Minnesota 
Mississippi 


Missouri 


Montana 


See notes at end of table. 


FY 1992 
62 


66 
n/a 
86 
71 
45 
40 
79 
85 
n/a 
53 
50 
50 
70 
66 
64 
28 
50 
44 
56 
53 


Secondary 
FY 2001 
63 


63 
87 
86 
75 
41 
42 
86 
n/a 
n/a 
53 
49 
50 
65 
60 
n/a 
56 
56 
49 
55 
50 
65 
71 
n/a 
36 
53 
71 
63 


FY 2010 
64 


68 
85 
84 
74 
38 
40 
81 
85 
81 
51 
50 
50 
65 
60 
64 
51 
50 
55 
56 
50 
65 
70 
60 
42 
53 
72 
65 


FY 1992 
38 


34 
n/a 
14 
29 
55 
60 
22 
15 
n/a 
47 
50 
50 
30 
34 
36 
72 
50 
56 
44 
47 
30 
19 
42 
91 
54 
30 
35 


Postsecondary 
FY 2001 
37 


37 
13 
14 
25 
59 
58 
14 
n/a 


n/a 


n/a 


FY 2010 
36 


32 
15 
16 
26 
62 
60 
19 
15 
19 
49 
50 
50 
35 
40 
36 
49 
50 
45 
44 
50 
35 
30 
40 
58 
47 
28 
35 


Change in 
percentage share 
allocated to 
secondary from 
FY 2001 through 
FY 2010 


1 
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Exhibit 2.14.— cont. 


Percentage of Perkins Title I formula-allocated funds allocated to the secondary and 
postsecondary levels, fiscal years 1992, 2001, and 2010, and change in the percentage share 
allocated to the secondary level, by state: FY 2001 through FY 2010 


State 
Nebraska 
Nevada 

New Hampshire 
New Jersey 
New Mexico 
New York 
North Carolina 
North Dakota 
Ohio 
Oklahoma 


Oregon 
Pennsylvania 
Rhode Island 
South Carolina 
South Dakota 
Tennessee 
Texas 

Utah 
Vermont 
Virginia 
Washington 
West Virginia 
Wisconsin 


Wyoming 


FY 1992 


50 


45 


n/a 


Secondary 
FY 2001 


60 
68 
79 
66 
36 
57 
n/a 
65 
82 
88 
n/a 
70 
n/a 
82 
43 
89 
57 
58 
80 
85 
43 
78 
44 
65 


FY 2010 


55 
68 
80 
55 
50 
52 
64 
65 
88 
84 
50 
70 
85 
67 
50 
85 
71 
60 
75 
85 
44 
71 
48 
60 


FY 1992 


50 
25 
n/a 
23 
92 
34 
31 
35 
18 
16 
50 
29 
11 
a) 
58 
14 
44 
40 
20 
15 
58 
23 
55 


n/a 


Postsecondary 
FY 2001 


40 
32 
21 
34 
64 
43 
n/a 
35 
18 
12 
n/a 
30 
n/a 
18 
57 
11 
43 
42 
20 
15 
57 
22 
56 
35 


FY 2010 


45 
32 
21 
45 
50 
48 
36 
35 
12 
16 
50 
30 
15 
33 
50 
15 
30 
40 
25 
15 
56 
29 
52 
40 


Change in 


percentage share 


allocated to 
secondary from 


FY 2001 through 


FY 2010 


42 


Exhibit reads: Alaska allocated 87 percent of local funds to the secondary level in FY 2001 and 85 percent in FY 2010, a decrease of 2 


percent. 


n/a not available or missing data. 


“The national average is the average proportion of Title I funds dedicated to each level across all states available in a given year (47 states in 


FY 1992; 44 states in FY 2001; and all U.S. states, including the District of Columbia, in FY 2010). 


SOURCE: Silverberg et al. (2004); and U.S. Department of Education, Office of Vocational and Adult Education, 2011, Internal Perkins 
Data Base System and State Consolidated Annual Report data system information for FY 2010. 
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Of the 44 states for which data from Perkins II and IV are available, the percentage of funds 
allocated to the secondary level differed by 5 percent or less in 33 states from FY 2001 to FY 
2010, with 13 states registering virtually no change (ie., differences of 1 percent or less). 
Among the eleven states with percentage point changes greater than 5 percent, six states in- 
creased the proportion of resources flowing to the secondary level, and five reduced the sec- 


ondary share of the funds. 


The case studies indicate that some states had recently changed the division of funds based 
on negotiations between representatives for the secondary and postsecondary levels, while 
other states maintained divisions that were determined many years ago. For example, negoti- 
ations within the last three years in one state resulted in a formula based on enrollments av- 
eraged over several years. In another state, the proportion allocated to each level had been 
negotiated in the early 1990s. One state-level administrator noted that the issue could be 
contentious: “I think the hardest part with Per&7ns, looking at it from a state level, is there 
seems to be more or less a constant battle between postsecondary and secondary on how 


much money they get.” 


In the survey, state directors indicated that they did not anticipate changes to the amounts al- 
located to the secondary and postsecondary levels for the remainder of Perkins IV.4° The 
majority (37) of the 50 secondary respondents anticipated no changes, and just one director 
reported an expected increase to the postsecondary amount; another 11 responded that they 
did not know. None of the 45 postsecondary respondents anticipated changes, and 12 re- 
ported that they did not know. 


Secondary Title | Local Allocations 


After the state has deducted administrative, leadership, and (if applicable) reserve fund re- 
soutces, remaining Title I funds are disbursed to eligible secondary and postsecondary pro- 
viders. Grant resources flowing into the secondary level are distributed to qualifying LEAs 
using a legislated distribution formula. The following section describes how states distribute 
these funds at the secondary level, and provides information on the number and characteris- 


tics of participating LEAs. 


Allocation criteria 
States allocate Title I secondary funds to eligible LEAs using a statutory distribution formula. 
The formula allocates 30 percent of funds based on the number of individuals aged 5-17 re- 
siding in the school district, and 70 percent based on the number aged 5-17 in the school 


40 Secondary and Postsecondary State Director Surveys, 2009. 
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district who lived in poverty during the preceding fiscal year.*! Calculations are based on all 
LEAs in the state eligible for Perkins IV resources. For example, an LEA serving 5 percent 
of those aged 5-17 living in poverty among all eligible districts in the state would be eligible 
to receive 5 percent of the 70 percent of formula-allocated funds. Because the formula dis- 
tributes resources based on the number of individuals residing within a district’s boundaries 
and not on the number participating in CTE programs, funding amounts are not tied to the 
presence or size of CTE programs offered within LEAs. Consequently, demographically 
similar districts may qualify for roughly equivalent amounts of Perkins IV funding irrespec- 


tive of student participation in CTE or the level of LEA investment in CTE programs. 


Perkins IV offers two alternatives to the legislated formula. First, states may apply for a waiv- 
er for more equitable distribution that allows states to use an alternative formula to distribute 
resources to local subgrantees.*#* This waiver may be granted only if an eligible agency can 
demonstrate that its use more effectively targets resources on the basis of poverty, as defined 
by the Office of Management and Budget. Second, if 15 percent or less of a state’s grant is 
made available to either the secondary or postsecondary level, the Special Rule for Minimal 
Allocation allows these funds to be distributed on a competitive basis or by an alternative 
method.* According to Department staff, no state used these options at the secondary level 
during the first four years of Perkins IV; the options also were available but unused for sec- 


ondary disbursement under Perkins III (Silverberg et al. 2004). 


As part of the allocation process, state agencies issue an annual application that LEAs seek- 
ing funds must complete and submit. To prepare the application, an LEA must develop a 
plan for CTE programs that indicates how it will use federal resources to address the re- 


quired activities detailed in Perkins IV and the state plan. 


Findings from the secondary state director survey indicated that, with few exceptions, states 
approved nearly all of the 4,060 secondary local subgrantee applications submitted across 45 
states for the 2008-09 program year.‘ States described working closely with LEAs to im- 
prove applications that did not initially meet Perkins requirements so that all or most appli- 
cants eventually qualified to receive funds. Among the 45 secondary state directors 
responding, only three reported rejecting funding applications, with one state accounting for 
23 of the 26 rejected applications. The odds that an LEA would qualify for funding, either in 


response to its original application or after revision and resubmission, approached 100 per- 


41 States must use either Census data used to determine district eligibilities under Title I of the E/emen- 
tary and Secondary Education Act (ESEA) or data collected by the National Center for Education Statis- 
tics (Sec. 131(a)). Poverty data are obtained from Sec. 1124(c)(1)(A) of ESEA. 

42 Sec. 131(c). 

#8 Sec. 133 (a). 

“4 Perkins IV information obtained through personal communication with Andrew Johnson, Grants 
Management Specialist at the Office of Vocational and Adult Education, April 18, 2011. 

45 Secondary and Postsecondary State Director Surveys, 2009. 
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cent. This finding parallels that reported in the previous national assessment, which found 
that 93 percent of local applications for Perkins secondary funds in FY 2001 were approved 
and awarded, with only six states rejecting one or more local applications (Silverberg et al. 
2004). 


State directors in the three states that rejected one or mote local applications offered several 
reasons for their decision not to award funding. These included failure on the part of the 

LEA to provide an adequate plan for allocating funds and collecting accountability data, fail- 
ure to provide past accountability data, poor past performance, insufficient program quality, 


ot an inability or unwillingness to join a consortium.‘ 


Percentage and type of LEAs awarded funding 
Although all LEAs enrolling students may qualify for a Perkins IV grant, resources typically 
are distributed to those offering secondary programs (1.e., where grades 10-12 are the high- 
est offered) (Exhibit 2.15). 


Exhibit 2.15. 
Percentage of LEAs with secondary schools awarded Title I funds: 
Fiscal years 1992, 2001, and 2010 


FY 1992 FY 2001 FY 2010 


LEAs with secondary school(s) awarded Title I funds 68 64 68 
Exhibit reads: Among the 39 states for which data were available in all three years, about 68 percent of LEAs with secondary 
schools received a Title I grant in FY 1992. 

NOTE: “LEAs with secondary schools,” as defined here and used as the denominator in these calculations, include all LEAs and 
identifiable CTE area schools that have at least 10th-, 11th-, or 12th-grade education, based on an analysis using the NCES 
Common Core of Local Education Agency Universe Survey: School Year 2007-08 (most recent year available). In FY 2010, these 
institutions accounted for about 95 percent of all grantees for which a CCD match was found. The analysis is based on the 39 states 
for which data were available in all three years. 

SOURCE: (1992 and 2001 data) Silverberg et al. (2004); (2010 data) Secondary State Director Survey Fiscal Data, 2009; and National 
Center of Education Statistics (NCES), Common Core of Local Education Agency Universe Survey: School Year 2007-08. 


An analysis of the 39 states for which data were available for FY 1992, FY 2001, and FY 
2010 revealed changes of only a few points in the percentage of districts receiving Title I 
funds during this period. From 64 to 68 percent of districts with secondary schools were al- 


located funds in each of the three years. 


The requirement that all grant recipients offer one or more POS to obtain funding does not 
seem to have adversely affected local CTE program administration and implementation. On 


a1 to 5 rating scale, with 1 being a “very negative impact” and 5 a “very positive impact,” 


46 Because responses to some question items were less than 85 percent, results should be interpreted 
with caution. 
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LEA survey respondents rated the impact of adopting or developing POS at just over 3, in- 
dicating that these policies had no or a slightly positive impact on their administration and 


implementation of CTE programs.*” 


In FY 2010, about 95 percent of Perkins IV subgrantees were in LEAs with a highest grade 
of 12, which received about 97 percent of the funds allocated. LEAs in which grade 8 was 

the highest grade offered accounted for about 4 percent of subgrantees, but received just 1 
percent of funds (Exhibit 2.16). 


Exhibit 2.16. 
Percentage of local secondary Perkins Title I allocations, by highest grade offered and type 
of LEA: FY 2010 


Highest grade and LEA type Number of grantees Percentage of grantees Percentage of funds 
Highest grade offered 


Grade 8 432 4 1 

Grade 12 9,205 95 97 

Other grades or not classified 91 1 2 
Type of LEA 

Local school district 9,520 98 94 

Regional education services agency 138 1 5 

Other* 70 1 4 


Exhibit reads: The highest grade offered by 432 of the LEAs that received a Title I grant in FY 2010 was grade 8. This type of LEA 
represented 4 percent of all grantees and received 1 percent of the funds allocated. 


“ Other includes administrative centets, institutions providing services to individuals with special needs, and charter school districts. 
NOTE: Analysis includes data for 9,728 LEAs in 49 states and the District of Columbia (excludes the 39 LEAs that received an allocation 
in FY 2010 but for which no NCES ID number or match was found). 

SOURCE: Secondary State Director Surveys Fiscal Data, 2009. 


Virtually all of the LEAs awarded funds were local school districts (98 percent). Regional 

education services agencies, which include Boards of Cooperative Educational Services, vo- 
cational school districts, and area CTE centers, also received Perkins Title I funds. Although 
these types of LEAs accounted for roughly 1 percent of the LEAs allocated funds, they re- 


ceived just over 5 percent of the funds allocated. 


Beginning with Perkins IT (FY 1991), Congress mandated that 1.25 percent of the total Per- 
kins appropriation be set aside for programs serving Native Americans and Alaska Natives, 
including schools funded by the Bureau of Indian Affairs (BIA).*8 Since Perkins II, BIA- 


“7 Secondary State Director Survey, 2009. Requirement to adopt at least one statewide POS to obtain 
funds, 3.46 (SE .026); Requirement to develop at least one POS to obtain funds, 3.39 (SE .025). 
48 Sec. 111(a)(1). 
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funded secondary schools have been ineligible to receive funds directly through this set- 
aside; these institutions are eligible, however, for formula-allocated Title I funds. Four states 
in the 2000-01 and 2008-09 program years reported awarding state Perkins funds to BIA 


secondary schools, either independently or as part of consortia.” 


Consortium provision 
To ensure that grants are adequate to support CTE programs and meet Perkins IV requite- 
ments, an LEA must be eligible to receive at least $15,000 using the basic grant distribution 
formula. If an LEA’s allocation does not meet this threshold, it may either join a consortium 
with one or more other LEAs, combining resources to meet the minimum allocation re- 
quirement, or seek a secondary consortia waiver allowing receipt of a grant less than 
$15,000.° 


States may grant waivers to any LEA that is either located in a rural, sparsely populated area 
ot is a public charter school offering secondary CTE programs, as long as the LEA can 
demonstrate that it is unable to join a consortium.>! Similar numbers of secondary state di- 
rectors reported issuing minimum local subgrantee waivers under Perkins II and Perkins IV 
(24 in the 2000-01 program year and 22 in the 2008-09 program year) (Exhibit 2.17).> 


Secondary State Director Survey, 2009. Because question items had response rates of less than 85 
percent, results should be interpreted with caution. 

50 Sec. 131(d). 

51 Sec. 131(d). Both the 2001 and 2009 state director surveys listed three reasons for waiving the min- 
imum allocation rule, including the inability to enter a consortium. Because inability to join a consor- 
tium is required of all waiver recipients, however, this section only reports two reasons for the LEA’s 
inability to join a consortium: location and charter status. 

>? The 2008-09 data may underestimate waiver use, because three states reporting the use of waivers 
did not provide information on the number and types issued. 
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Exhibit 2.17. 
Minimum allocation waiver use at the secondary level: 
Program years 2000-01 and 2008-09 


Number and type of waivers 2000-01 2008-09* 
Total number of waivers reported 434 286 
Range (among states) 1-79 1-152 
Number of states reporting waivers 24 22 
Waiver type 
Sparsely populated area (number of waivers) 275 269 
Number of states 22 19 
Public charter school (number of waivers) 48 12 
Number of states 1 2 


Exhibit reads: In 2000-01, 24 states reported issuing 434 consortia waivers. Twenty-two states reported issuing a total of 275 waivers to 
LEAs located in sparsely populated areas. 


“Three of the states reporting waivers in 2008-09 did not know the number and types of waivers received. 
SOURCE: White et al. (2004); and Secondary State Director Survey, 2009. 


In both years, states gave the largest number of waivers to LEAs located in sparsely populat- 
ed (rural) areas that precluded them from joining a consortium. A total of 19 states granted 
such waivers in the 2008-09 program year, with 16 states issuing waivers to 10 or fewer 
LEAs.* In contrast, one predominantly rural state granted 152 waivers, one to every LEA in 
the state. States were less likely to take advantage of the waiver for public charter schools: 


only two states issued a total of 12 such waivers in 2008-09. 


Rather than receiving a waiver, the majority of LEAs receiving grants of less than $15,000 
joined consortia. In FY 2010, 32 of the 50 states that submitted fiscal data funded at least 


one consortium, and 22 states funded 10 or more (Exhibit 2.18). 


Nationwide, about 5,570 LEAs received Perkins funds through 703 consortia, representing 
about 22 percent of the allocation amounts to secondary subgrantees reported by the 49 
states for which FY 2010 data were available. The number of LEAs participating in consortia 
varied widely across states, with some states funding as few as two LEAs through consortia 
and others 500 or more. Although the legislation does not limit Title I consortia members to 
one education level, consortia members in the case study states with Title I consortia were all 


at the secondary level. 


Grants for all participating subgrantees within a consortium are combined, and members 
must agree on how to allocate the aggregated resources to support CTE services across the 
consortium. One state director noted that LEAs had largely ignored this requirement in the 


past and reallocated funds to consortium members to be spent individually. Under Perkins 


53 Data not shown. 
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IV, however, consortia members within this state are meeting this requirement. Another 
state-level administrator reported that, in an effort to encourage collaboration, the state had 
imposed a requirement that LEAs could form a consortium only if they joined with a re- 
gional education services unit or community college. As a result, federal funds were more 
likely to be invested in creating high-quality CTE programs, rather than used as a revenue 
stream for LEAs to make equipment purchases. Another state administrator reported that 
the state’s consortia facilitate disseminating information and aid in organizing and providing 


professional development. 
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Exhibit 2.18. 
Number of secondary Title I consortia allocations, number of LEAs participating in 
consortia, and average consortia allocation amount, by state: FY 2010 
Total number of Total number of LEAs Avetage consortia 

State consortia grants participating in consortia gtant (amount) 
Alabama 2, 6 84,300 
Alaska 0 0 0 
Arizona 0 0 0 
Arkansas 16 179 218,700 
California 20 102 101,800 
Colorado 14 108 50,900 
Connecticut 6 28 63,300 
Delaware n/a n/a n/a 
District of Columbia 0 
Florida 0 0 0 
Georgia 0 0 0 
Hawaii 0 0 0 
Idaho 18 69 43,800 
Illinois 52 483 303,300 
Indiana 48 291 418,900 
Towa 49 327 51,500 
Kansas 18 191 69,500 
Kentucky 5 12 23,700 
Louisiana 0 0 0 
Maine 0 0 0 
Maryland 0 0 0 
Massachusetts 2 11 147,000 
Michigan 24 523 679,800 
Minnesota 26 337 240,200 
Mississippi 20 57 110,600 
Missouri 60 418 187,400 
Montana 0 0 0 
Nebraska 15 233 79,700 
Nevada 0 0 0 
New Hampshire 17 28 250,700 
New Jetsey 1 2 27,900 
New Mexico 19 42,700 
New York 0 0 
North Carolina 0 0 0 
North Dakota 26 151 46,900 


See notes at end of table. 
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Exhibit 2.18.—cont. 
Number of secondary Title I consortia allocations, number of LEAs participating in 
consottia, and average consortia allocation amount, by state: FY 2010 


Number of Number of LEAs Avetage consottia 

State consortia grants participating in consortia gtant (amount) 
Ohio 0 0 $0 
Oklahoma 59 311 44,700 
Oregon 15 142 201,700 
Pennsylvania 72 462 255,800 
Rhode Island 10 38 397,200 
South Carolina 0 0 0 
South Dakota 21 140 38,900 
‘Tennessee 1 25 1,582,900 
Texas 52 554 85,700 
Utah 1 2 15,700 
Vermont 0 0 0 
Virginia 1 2: 278,700 
Washington 1 2 16,300 
West Virginia 0 0 0 
Wisconsin 30 317 134,300 
Wyoming 0 0 0 
Puerto Rico n/a n/a n/a 
Total 706 5,570 n/a 


Exhibit reads: In FY 2010, Alabama had 2 Title I secondary consortia with a total of 6 LEA members, and the average allocation 
made to consortia in Alabama that yeat was $84,300. 
n/a Data not available. 


“ Average grant is calculated among states awarding secondary Title I consortia grants. 
SOURCE: Secondary State Director Surveys Fiscal Data, 2009. 


Number and size of local allocations 
Nationally, the number of LEA subgrantees differed little (by less than 1 percent) between 
Perkins III and Perkins IV, reflecting the consistency of the distribution formula across the 
two Acts. The 39 states for which multi-year data were available reported a total of 4,388 al- 
location amounts to individual LEAs and consortia in FY 2010, compared to 4,424 grants in 
FY 2001 and 4,232 in FY 1992 (Exhibit 2.19). 
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Exhibit 2.19. 


52 


Number and average amount of Title I local secondary allocations, fiscal years 1992, 2001, 


and 2010, and difference and percent change: FY 2001 through FY 2010 


Difference 
from 

FY 2001 

Number and through 

amount of allocations FY 1992 FY 2001 FY 2010 FY 2010 

Number of allocations 4,232 4,424 4,388 -36 
Amount (average allocation size)" 

Nominal dollars 76,238 101,813 112,934 11,121 

Real dollars (2010) 118,629 125,505 112,934 -12,571 


Percent 


change from 


FY 2001 
through 
FY 2010 


-1 


11 
-10 


Exhibit reads: There were 4,424 Title 1 secondary allocations in 2001 and 4,388 in FY 2010, a decrease of 36 grants or about 1 percent. In 


nominal dollars, the average allocation in FY 2001 was $101,813, and in 2010, $112,934, an increase of $11,121 or 11 percent. 


“Current dollars are the actual amounts as they were allocated and reported in the given year. Real dollars have been adjusted for inflation. 


NOTE: Calculations include the 39 states for which secondary local allocation data were available for all three years. 


SOURCE: Silverberg et al. (2004); NACTE Secondary State Director Survey Fiscal Data, 2009; and Real dollar calculations: Bureau of 


Labor Statistics, Inflation Calculator, http://data.bls.gov/cgi-bin/cpicalc.pl (accessed January 20, 2011). 


The average LEA allocation increased by roughly one-third between Perkins II and Perkins 


II. As documented in the previous national assessment, this increase is likely due to the con- 


fluence of several factors, including the increase in appropriations between Perkins II and 


Perkins III and a change in the proportion of resources distributed to local providers (Silver- 


berg et al. 2004). In keeping with past trends, the average allocation amount also increased 


between Perkins HI and IV, climbing from $101,813 in FY 2001 to $112,934 in FY 2010. In 


addition to the factors noted above, the increase in the average allocation may reflect addi- 
tional Title I resources in states that consolidated Title I and H (Tech Prep) funds. This in- 


crease represents a nominal gain of about $11,121, but in real 2010 dollars the average 


allocation declined by more than $12,000 between these years. 


In FY 2001 and FY 2010, the average Perkins IV LEA allocation varied widely by state, and 


ranged from $2,562,592 to $17,097 in FY 2010 (Exhibit 2.20).54 Among the 42 states for 


which FY 2001 and FY 2010 data were available, 30 reported an increase in the average LEA 


allocation, and 12 reported a decrease. Variations in both the size of the average allocation 


across states and the average allocation within states over time can be attributed to changes 


in the proportion of funds allocated to the secondary and postsecondary levels, the number 


of LEA applicants, the proportion of LEAs in consortia, relative LEA setvice atea popula- 


tion and poverty levels, and average LEA size. 


54 In some states, allocation amounts for 2010 are estimates based on state formula calculations. Actu- 


al grant amounts disbursed to recipients may vary. 
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Average, maximum, and minimum Title I secondary local allocation amounts awarded and 


direction of average allocation amount change, by state: FY 2001 and FY 2010 


Change 
Average allocation amount from Maximum allocation 

State FY 2001 FY 2010 FY 2001 FY 2001 FY 2010 

National" $101,813 $112,934 x $13,526,258 $13,700,000 
Alabama 75,837 85,413 + 1,211,668 1,195,365 
Alaska 66,078 56,574 - 1,102,628 953,409 
Arizona 120,961 158,091 + 2,730,984 3,480,298 
Arkansas 87,356 106,196 + 497,306 623,111 
California 118,929 122,976 + 7,632,316 7,023,392 
Colorado 75,131 84,624 + 811,763 933,633 
Connecticut 77,709 79,346 + 1,048,221 776,539 
Delaware n/a n/a n/a n/a n/a 
District of Columbia n/a 600,000 n/a n/a 2,528,015 
Florida 324,189 421,469 + 3,969,864 4,414,487 
Georgia 72,072 97,409 + 818,653 1,042,007 
Hawaii” 2,285,140 2,562,592 + 2,285,140 2,562,592 
Idaho 62,024 62,190 + 355,505 370,807 
Illinois 354,369 56,590 - 8,511,898 8,246,886 
Indiana n/a 418,916 n/a n/a 1,211,526 
lowa n/a 60,978 n/a n/a 476,836 
Kansas 71,443 44,663 = 650,327 712,416 
Kentucky 48,499 49,060 + 1,084,916 1,284,610 
Louisiana 149,794 142,170 - 1,469,956 776,767 
Maine 75,943 88,336 + 173,441 221,635 
Maryland 356,424 387,762 + 2,735,562 2,105,583 
Massachusetts n/a 156,502 n/a n/a 1,740,285 
Michigan n/a 832,825 n/a n/a 4,505,403 
Minnesota 104,482 240,150 + 1,045,389 741,180 
Mississippi 48,372 53,701 + 473,279 989,002 
Missouri 151,026 183,308 + 1,563,874 1,521,581 
Montana 26,224 17,097 - 656,347 280,412 
Nebraska 79,984 98,607 + 894,496 926,719 
Nevada 225,457 297,722 + 2,181,208 3,390,042 
New Hampshire 136,751 250,701 + 514,526 772,281 
New Jersey 112,699 46,459 - 982,291 923,120 
New Mexico 117,303 102,683 - 810,549 976,348 
New York 558,769 431,781 - 13,526,258 13,700,000 
North Carolina n/a 176,172 n/a n/a 1,681,133 


See notes at end of table. 


Minimum allocation 


FY 2001 
$582 


8,159 
15,000 
1,150 
15,379 
2,400 
3,934 
15,000 
n/a 
n/a 
13,030 
8,701 
2,285,140 
1,687 
31,392 
n/a 
n/a 
5,879 
2,702 
16,014 
27,900 
30,546 
n/a 
n/a 
15,001 
4,896 
3,222 
1,000 
15,235 
2,912 
45,228 
2,550 
22,777 
73,257 


n/a 


FY 2010 
$157 


15,534 
15,000 
1,581 
17,651 
1,891 
1,119 
13,643 
n/a 
54,161 
14,030 
2,831 
2,562,592 
4,058 
382 
51,866 
6,428 
819 
1,988 
12,245 
8,367 
35,583 
14,327 
95,807 
36,962 
4,922 
3,282 
1,000 
16,167 
2,039 
28,418 
3,776 
21,722 
21,528 
11,788 
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Exhibit 2.20.—cont. 
Average, maximum, and minimum Title I secondary local allocation amounts awarded and 
direction of average allocation amount change, by state: FY 2001 and FY 2010 


Change 
Average grant amount from Maximum gtant Minimum grant 
State FY 2001 FY 2010 FY 2001 FY 2001 FY 2010 FY 2001 FY 2010 
North Dakota $50,279 $62,391 + $209,796 $244,408 $5,547 $14,712 
Ohio 322,241 332,428 + 2,667,048 2,340,533 46,833 42,768 
Oklahoma 78,113 58,283 - 994,299 1,149,840 3,986 1,165 
Oregon n/a 158,727 n/a n/a 649,175 n/a 1,499 
Pennsylvania 213,181 263,031 + 5,704,966 6,558,290 19,543 11,332 
Rhode Island n/a 397,188 n/a n/a 1,247,770 n/a 104,240 
South Carolina 128,254 157,825 + 933,272 1,085,453 13,588 16,537 
South Dakota 28,605 43,389 + 146,032 223,481 886 3,115 
Tennessee 139,909 151,613 + 3,086,573 3,013,420 11,744 8,267 
Texas 41,164 137,116 + 3,104,501 3,798,821 1,500 1,597 
Utah 688,846 188,152 - 2,418,246 1,066,222 225,604 11,360 
Vermont 166,944 159,369 - 290,948 370,720 12,843 39,000 
Virginia 137,260 140,565 + 1,420,159 1,607,901 5,640 5,039 
Washington 34,955 32,866 - 543,439 501,723 582 157 
West Virginia 90,367 90,904 + 505,548 510,617 8,779 11,995 
Wisconsin 92,334 110,510 + 2,212,247 2,156,789 5,151 17,170 
Wyoming 47,060 51,564 + 332,601 362,856 2,496 4,075 
Puerto Rico n/a n/a n/a n/a n/a n/a n/a 


Exhibit reads: In Alabama, the average Title I secondary allocation was $75,837 in FY 2001 and $85,413 in FY 2010. The maximum and 
minimum allocations for this state in FY 2010 were $1,195,365 and $15,534, respectively. 

n/a Not available or missing data. 

+ Average allocation size increased from FY 2001 through FY 2009. 

— Average allocation size decreased from FY 2001 through FY 2009. 

a National average reflects the trend calculations in Exhibit 2.19. 

b Hawaii has just one school district that receives all of the state’s Title I Perkins funds allocated to the secondary level. 

NOTE: Allotment amounts for FY 2010 for some states are estimates based on formula calculations. Actual allocations may vary according 
to consortia memberships. Amounts may also include carry over funds, which are unspent local funds from the previous year that are 
reallocated by formula. 

SOURCE: Silverberg et al. (2004); and Secondary State Director Survey Fiscal Data, 2009. 


Allocations by LEA poverty level, location, and size 
Several aspects of the legislation target Perkins resources to specific groups of students and 
types of LEAs. As in previous legislation, Perkins IV has a compensatory emphasis. As out- 
lined earlier, Perkins IV allocates 70 percent of secondary funds according to the relative 
number of low-income individuals in a community and the remaining amount according to 
the relative percentage of residents aged 5-17. Through the reserve allocation, states also 
have some discretion to direct additional resources to rural and sparsely populated areas and 


to areas with high numbers or high percentages of CTE students. The analyses presented 
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here provide an overview of how the various allocation criteria distributed funds by LEA 


poverty level, size, and location.* 


LEA poverty level 
In both FY 2001 and FY 2010, LEAs received an average of about $40 per secondary stu- 
dent (Exhibit 2.21). This average differed across low-, medium-, and high-poverty LEAs in 
FY 1992, FY 2001, and FY 2010. 


In high-poverty LEAs, the funding level per secondary student was $53 in both FY 2001 and 
FY 2010, a higher level of funding than in medium- and low-poverty LEAs in both years. 
High-poverty LEAs did receive a lower proportion of Perkins funds in FY 2010 than in FY 
2001 (36 vs. 42 percent), but they represented about the same proportion of subgrantees (28 
percent). These patterns may reflect demographic shifts and changes in the mix of poverty 
levels among the districts applying for Perkins funds and the distribution of reserve funds, 
which states may have allocated in ways that disproportionately benefit high-poverty LEAs 


on a per student basis. 


Exhibit 2.21. 
Average Perkins Title I allocations pert secondary student and distributions of funds and 
local subgrantees, by LEA poverty level: Fiscal years 1992, 2001, and 2010 


FY 1992 FY 2001 FY 2010 
Perkins Perkins Perkins 
dollar dollar Percent of dollar Percent of 
amount per amount per funds amount per funds 
secondaty secondaty received by Percent of secondaty received by Percent of 

LEA poverty level” student student grantees grantees student grantees grantees 
High-poverty $51 $53 42. 28 $53 36 28 
Medium-poverty 28 32 42. 56 37 51 55 
Low-poverty 32 41 16 16 25 13 18 
All LEAs 32 40 29 


Exhibit reads: In FY 2001, high-poverty LEAs received an average of $53 in Perkins funding per secondary student. These types of LEAs 
received 42 percent of all local secondary Title I funds at the secondary level, and they accounted for about 28 percent of the secondary 
Title 1 grantees. 


“Poverty levels are based on quartiles determined by the percentage of students within a district eligible for free or reduced-priced lunches. 
For FY 1992 and FY 2001: Low = 9 percent ot less, Medium = 10 to 49 percent, High = 50 percent or more. For FY 2009: Low = 25 
percent or less, Medium = 26 to 50 percent, High = 59 percent or more. 

NOTE: Calculations are based on the 29 states for which trend data are available. The 2 percent of LEAs for which CCD enrollment 
numbers are unavailable are excluded; these include BOCES in New York and joint vocational school districts in Ohio. The unit of analysis 
is a secondary student (grades 9-12) because most take at least one vocational education course. Therefore, the Perkins dollar amounts in 
this table indicate the amount of federal money that is calculated per potential participant in secondary vocational education. Detail may not 
sum to 100 percent because of rounding. 

SOURCE: Silverberg et al. (2004); Secondary State Director Survey Fiscal Data, 2009; and National Center for Education Statistics (NCES) 
Common Core of Data (CCD) Elementary/Secondary School Universe Survey: School Year 2007-08. 


55 The trend analyses reported in this section are based on the 29 states for which data were available 
for FY 1992, FY 2001, and 2010. Data for the 49 states (and the District of Columbia) for which data 
ate available in 2010 are included in the exhibits, but not discussed in the text. 
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LEA location 
In both FY 2001 and FY 2010, more than half of LEA subgrantees were located in rural are- 
as (Exhibit 2.22). Because of their relatively small enrollments, however, these LEAs re- 
ceived less than one-fourth of Title I secondary funds (about 24 and 21 percent of funds in 
FY 2001 and FY 2010, respectively). By contrast, urban districts accounted for about 7 per- 


cent of subgrantees in both years, but they received about 38 and 32 percent of Perkins funds 
in FY 2001 and FY 2010, respectively. 


Exhibit 2.22. 
Average Perkins Title I allocations per secondary student and distribution of funds and local 
subgrantees, by LEA locale: Fiscal years 1992, 2001, and 2010 


FY 1992 FY 2001 FY 2010 
Perkins Perkins Percent of Perkins Percent of 
dollar amount dollar amount funds dollar amount funds 

LEA per secondary persecondary teceivedby Percentof persecondary received by Percent of 
locale* student student grantees grantees student grantees grantees 
Urban $43 $50 38 8 $45 32 7 
Suburban 24 30 38 23) 32 47 40 
Rural 39 54 24 59 53 21 54 
All LEAs 32 40 39 


Exhibit reads: In FY 2001, urban LEAs received an average of $50 per secondary student in Perkins funds. These types of LEAs received 
about 38 percent of Perkins ‘Title I local funds granted at the secondary level and accounted for 8 percent of all secondary grantees. 

“The CCD classifies LEA locales as either an LEA that primarily serves a central city of a Metropolitan Statistical Area (MSA); an LEA that 
serves an MSA but not its central city (suburban); or an LEA that does not serve an MSA (tural). Since 2005—06 was the last year this 
classification scheme was used, the 2009 data were analyzed using CCD LEA-level data from that year. 

NOTE: Calculations are based on the 29 states for which data were available in all three years and in which more than 80 percent of 
grantees in a state had an NCES ID. The 2 percent of LEAs for which CCD enrollment numbers are unavailable are excluded in FY 2010; 
these include BOCES in New York and joint vocational school districts in Ohio. The unit of analysis is a secondary student (grades 9-12) 
because most take at least one vocational education course. Therefore, the Perkins dollar amounts in this table indicate the amount of 
federal money that is calculated per potential participant in secondary vocational education. Detail may not sum to totals because of 
rounding. 

SOURCE: Silverberg et al. (2004); Secondary State Director Survey Fiscal Data, 2009; National Center for Education Statistics (NCES) 
Common Core of Data (CCD) Local Education Agency Universe Survey: School Year 2005-06; and NCES CCD Elementaty/Secondary 
School Universe Survey: School Year 2007-08. 


As in FY 2001, Perkins funds in FY 2010 disproportionately benefited rural and urban LEAs 
on a per student basis. These LEAs received an average of $53 and $45 per secondary pupil, 
respectively. By contrast, per pupil funding in suburban LEAs was $32. The relatively higher 
levels of per pupil funding for rural LEAs in FY 2001, compared with FY 1992, resulted 
from the introduction of the reserve fund in Perkins HIT, which was intended partly to in- 
crease assistance for rural CTE programs (Silverberg et al. 2004). This policy continues to 
have a salutary effect on rural funding levels under Perkins IV. 
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LEA size 
Medium-sized LEAs had the largest proportion of subgrantees in FY 2010 (42 percent), but 
LEAs enrolling 3,000 or more students received the largest proportion of funds (84 percent). 
LEAs enrolling fewer than 500 students received a small proportion of overall Perkins funds 


(2 percent), but they accounted for 25 percent of all subgrantees (Exhibit 2.23). 


Exhibit 2.23. 
Average Perkins Title I allocations per secondary student and distribution of funds and local 
subgrantees, by subgrantee size: Fiscal years 1992, 2001, and 2010 


FY 1992 FY 2001 FY 2010 
Perkins Perkins Percent of Perkins Percent of 
dollar amount dollar amount funds dollar amount funds 
per secondary persecondary received by Percent of pet secondary received by Percent of 
LEA size student student subgrantees subgrantees student subgrantees subgrantees 
Large $32 $37 79 34 $38 84 32 
(3,000 or more 
students) 
Medium 32 53 18 45 45 15 42 
(500 to 2,999 
students) 
Small 37 88 3 21 50 2 25 
(less than 500 
students) 
All school 
districts 32 40 38 


Exhibit reads: In FY 2001, large LEAs received an average of $37 per secondary student in Perkins funds. These types of LEAs received 
about 79 percent of Perkins Title I local funds granted at the secondary level and accounted for about 34 percent of all secondary subgrantees. 
NOTE: Calculations are based on the 29 states for which data were available in all three years and in which more than 80 percent of 
subgrantees in a state had an NCES ID. The 2 percent of LEAs for which CCD enrollment numbers are unavailable are excluded in FY 2010; 
these include BOCES in New York and joint vocational school districts in Ohio. The unit of analysis is a secondary student (grades 9-12) 
because most take at least one vocational education course. Therefore, the Perkins dollar amounts in this table indicate the amount of federal 
money that is calculated per potential participant in secondary vocational education. Detail may not sum to totals because of rounding. 
SOURCE: Silverberg et al. (2004); Secondary State Director Survey Fiscal Data, 2009; NCES Common Core of Data (CCD) Local Education 
Agency Universe Survey: School Year 2007-08; and NCES CCD Elementary/Secondary School Universe Survey: School Year 2007-08. 


On a per secondary student basis, however, allocations to small districts were equivalent to 
an average of $50 per student in FY 2010, or about 32 percent more than the average $38 
that large districts received. Funding levels per secondary student in FY 2010 were smaller in 
small- and medium-sized LEAs than they were in FY 2001; for example, small districts were 
allocated $88 per student in FY 2001 and $50 per student in FY 2010. Per student alloca- 


tions in large LEAs were about the same across the two years. 
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Use of secondary funds at the local level 
Guidelines for the use of funds at the local level are divided into 9 required and 20 permis- 
sive uses. All of the eight required uses in Perkins II] were maintained in Perkins IV, with only 
some textual changes made to the legislation. These changes added additional detail or speci- 
ficity, but they did not change the substance of the required use. For example, specifications 
such as the “effective use of scientifically based research and data to improve instruction” 
were added to the requirement to provide professional development to faculty, administra- 
tors, and counselors. Mention of “Tech Prep” was removed from the requirement to link 
secondary and postsecondary CTE programs, and a requirement to offer POS was added. 
The only new required use in Perkins IV was to provide activities to prepare special popula- 
tions enrolled in CTE programs for high-skill, high-wage, or high-demand occupations lead- 
ing to self-sufficiency (Exhibit 2.24).°° 


Fifteen of the 20 permissive local uses of Perkins IV’ funds also had parallels in Perkins II, 
although several uses acquired new language or specifications.*” The changes include provid- 
ing work-related experiences to teachers and faculty as well as students, and addressing the 
integration of academic and CTE education in CTE teacher preparation programs. Perkins 
IV also added six new permissive uses, including support for entrepreneurship education, 
career-themed learning communities, training programs in automotive technologies, and 


pooling funds with one or more eligible recipients for innovative initiatives. 


© Sec. 135(b). 
57 Sec. 135(c). 
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Exhibit 2.24. 
Summaty of required uses of local funds: Perkins IIT and IV 


Required uses of local funds Perkins [iI Perkins IV 
Strengthen academic and career and technical skills of CTE students through the integration 
of academics with CTE programs through a coherent sequence of courses, such a career and 


technical POS described in the state plan, to ensure learning in core academic subjects as ¥ v 
defined by ESEA, and CTE subjects.* 

Provide students with strong experience in and understanding of all aspects of an industry, 7 V 
including work-based learning experiences.* 

Develop, improve or expand the use of technology in CTE.* Vv v 
Professional development consistent with the state plan to secondary and postsecondary 

teachers, faculty, administrators, and career guidance and academic counselors who are Vv Vv 
involved in integrated CTE programs.* 

Evaluations of vocational and technical education programs being carried out with Perkins 

funds, including an assessment of how the needs of special populations are being met. Vv Vv 
Initiate, improve, expand and modernize quality vocational and technical education programs, F y 
including relevant technology.* 

Provide services that ate of sufficient size, scope and quality to be effective. Vv v 
Link secondary and postsecondary CTE programs, including by offering the relevant elements 

of not less than one career and technical program of study described in the state plan.* v v 
Activities to prepare special populations, including single parents and displaced homemakers 

who are enrolled in CTE programs, for high-skill, high-wage or high-demand occupations Vv 


that will lead to self-sufficiency. 

Exhibit reads: Perkins IIT had 8 required uses for local funds. Six of these had changes in language, section details, or both in 
Perkins IV, which also added one required use. 

* Focus of activity was the same in Perkins III and IV, but language, section details, or both changed. 

NOTE: For full description of required local uses, see Perkins IV , Sec. 135(b). 

SOURCE: Perkins IV’, Sec. 135(b). 


According to local directors, the most common permissive uses were leasing, purchasing, 
upgrading, or adapting equipment (70 percent of LEAs) and providing career guidance and 
academic counseling (68 percent) (Exhibit 2.25).58 More than half of LEAs (58 percent) de- 
voted funds to support the implementation of POS. Less than one-fifth of LEAs, by con- 
trast, offered continuing education or job referral services (13 percent) or the newly 
permitted entrepreneurship education and training (18 percent). Districts are generally una- 
ble to provide information on the percentage or amount of funds spent on various uses be- 


cause funding sources may be combined and few states collect these data (Klein 2001). 


8 LEA Survey, 2009. All comparisons of the IHE and LEA survey data were tested for statistical sig- 
nificance using the Student’s ¢statistic, and all differences cited are statistically significant at the p < 
.05 level. 
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Exhibit 2.25. 


Percentage of local secondary grant recipients who reported using Title I funds for 


permissive uses: Program year 2008-09 


Percentage of 


Permissive use LEAs 
Leasing, purchasing, upgrading, or adapting equipment 70 
Providing career guidance and academic counseling 68 
Implementing CTE programs of study (POS) 58 
Promoting work-related experiences for students 56 
Providing programs for special populations a6) 
Assisting CTE student organizations 43 
Supporting nontraditional training and activities 39 


Involving business and labor in designing, implementing, 


and evaluating CTE programs covered by the Act 37 
Developing new CTE courses 36 
Supporting teacher preparation programs 39 
Promoting industry experiences for teachers 35 
Improving accountability data collection and reporting 30 
Offering mentoring and related support services 28 
Providing training programs in automotive technologies 23 
Entrepreneurship education and training 18 
Offering continuing education or job referral services 13 
Other" 4 


60 


Exhibit reads: In program year 2008-09, 70 percent of LEAs reported using Perkins funds for leasing, purchasing, upgrading, or adapting 


equipment. 


“Other” includes CTE teacher training and professional development, career academies, integration of academics, literacy, technology, 


career software, industry exams, and inquiry-based learning. 
NOTE: Estimates were weighted to reflect population values. 


SOURCE: LEA Survey, 2009. 


Case study evidence indicates that within-district allocations were made chiefly by the local 
director on the basis of programmatic priorities and perceived need and the formality of the 
funding allocation process varied by district. In some districts, allocation decisions were in- 
formed by meetings and discussions with faculty members, whereas others used a written 


application process requiring programs to document how funds would be used. 


In the survey and case study visits, local directors reported equipment purchases, particularly 
for software associated with specific fields such as business and computer-aided design, as 
their most common use of the funds. To support POS, sites reported using funds for pro- 
fessional development for faculty (including conference attendance) and consultants to assist 
with POS development, articulation agreements with postsecondary institutions, and indus- 


try certifications. Other uses included salaries for counselors, data specialists, or faculty; 
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sending students to conferences and other professional events; and industry certification ex- 


am fees for students. 


CTE is often more costly to provide than other types of instruction, chiefly because CTE 
classes tend to be smaller due to instructional and safety concerns, and salary and benefits 
costs ate accordingly higher. Specialized equipment and faculty training also contribute to 
CTE program costs (Klein 2001). Case study evidence suggests that local Perkins funds assist 
with the equipment and training purchases needed for program innovation. Citing the Per- 
kins requirement that funds should supplement, and not supplant, non-federal CTE funds, 
several local directors and faculty reported using Perkins funds to keep their programs up-to- 
date and in compliance with evolving industry standards. >? Several directors noted that for 
some programs, Perkins funds were the only resources available for these purposes. One di- 
rector explained, “...we do use Perkins money to develop new programs, to buy the equip- 
ment...that’s the only way we’ve ever had in the past to keep up with changes in technology 


because we have no other funds in [this state].” 


Postsecondary Title | Local Allocations 


As at the secondary level, nonreserve Title I funds are allotted on the basis of a statutory 
formula. In contrast to the secondary formula, however, which is based on population and 
poverty measures unrelated to CTE enrollments, the postsecondary formula includes a 
measure of CTE enrollment. States also are permitted to use alternative methods for distrib- 
uting Perkins IV funds through the special rule for minimum allocation and the waiver for 


mote equitable distribution. 


Allocation criteria 
The statutory formula allocates funds to eligible IHEs based on the subgrantee’s relative 
share of Pell Grant and BIA assistance recipients enrolled in CTE programs. As at the sec- 
ondary level, states can use an alternative formula if it more effectively targets funds to eco- 
nomically disadvantaged individuals.*! In FY 2001, 11 states received a formula waiver for 
more equitable distribution at the postsecondary level (Silverberg et al. 2004), and 9 states 
did so in FY 2010.@ 


»° Sec. 311 (a) 

69 Sec. 135(a). 

6! Sec. 132(a) 

62 PY 2010 information on formula waivers was gathered from Perkins 2008-2013 5-year state plans 
archived at U.S. Department of Education, Office of Vocational and Adult Education, 2011, Internal 
Perkins Database System and State Consolidated Annual Report Data System. 
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Punds can also be allocated on a competitive basis or through an alternative method if a 

state allocates 15 percent or less of local Title I funds to the secondary or postsecondary lev- 
els.°? Although no state used the special rule for minimal allocation at the secondary level in 
FY 2010, according to federal data on Perkins allocations, seven states used this option at the 


postsecondary level. 


States have different rationales for choosing to allocate less than 15 percent to the postsec- 
ondary level. In one case study, the director reported that though the state recognizes the 
importance of postsecondary CTE, it opted to continue past practice by using a competitive 
grant process that the state had pioneered. The means of allocating funds varied. For exam- 
ple, one state reported using a formula that targeted adult programs, and another allocated 
one-half on the basis of the legislated formula, and the other half using a formula based ex- 


clusively on local poverty levels. 


The postsecondary allocation process is similar to that used at the secondary level. Each 
year, state agencies issue an application that IHEs seeking federal funds must complete and 
submit. To apply, IHEs must develop a local plan for CTE programs that describes how re- 
soutces will be used to address the required activities detailed in Perkins IV and the state 


plan. 


Findings from the state postsecondary director survey indicated that states approved nearly 
all IHE grant applications submitted for the 2008-09 program year. Specifically, among the 
47 states responding, all but two of 1,179 applications submitted in FY 2009 were approved, 
either as originally submitted or after they were revised and resubmitted. This disapproval 
rate was similar to that reported in FY 2001, when three states rejected one or more JHE 


applications (Silverberg et al. 2004). 


The two state directors who reported rejecting a local application indicated that the rejec- 
tions were for reasons including failure to submit accountability data previously, poor past 
performance, low-quality programs, or an inability to meet the $50,000 minimum allocation 
threshold. 


Percentage and types of IHEs awarded funds 
Postsecondary grant recipients must be public or private nonprofit IHEs providing no less 
than a two-yeat program of instruction that can be credited toward a bachelor’s degree, trib- 


ally controlled colleges and universities, or nonprofit educational institutions (including 


63 Sec. 133(a). 

64 FY 2010 information on the special rule for minimal allocations was gathered from Perkins 2008— 
2013 5-year state plans and Perkins FY 2010 state budgets archived at U.S. Department of Education, 
Office of Vocational and Adult Education, 2011, Internal Perkins Data Base System and State Consol- 
idated Annual Report Data System. 
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LEAs and area CTE centers) that offer certificate or apprenticeship programs at the post- 
secondary level.® While participation is open to a range of institution types, the majority of 


postsecondary subgrantees were public two-year colleges in FY 2010 (Exhibit 2.26). 


Exhibit 2.26. 
Percentage distributions of postsecondary Title I subgrantees and funds, by education level 
and higher education subgrantee type: FY 2010 


Percentage Percentage of Perkins Title I 

Institution and program type of subgrantees postsecondary funds 
Total funds 100 100 
Programs for adults in non-IHE settings* 14 4 
Institutions of higher education 86 96 
All institutions of higher education subgrantees 100 100 
Administrative units” 2 9 
Public four-year 10 9 
Private nonprofit four-year 1 0 
Public two-year 83 80 
Private nonprofit two-year 1 1 
Public less-than-two-year 3 1 


Exhibit reads: Programs for adults in non-IHE settings accounted for about 14 percent of Title 1 local postsecondary subgrantees in FY 
2010 and received 4 percent of Title I local funds. 


“The majority of these subgrantees are area career and technical education centers that also serve secondary students, such as BOCES in 
New York. 


> Administrative units are offices for multi-campus college systems. 
NOTE: Calculations are based on the 50 states that submitted FY 2010 fiscal data. 
SOURCE: Postsecondary State Director Survey Fiscal Data, 2009; and NCES, Integrated Postsecondary Education Data System (IPEDS), 


Local allocation data revealed that about 14 percent of subgrantees were adult programs in 
secondary institutions, and 86 percent were postsecondary institutions. Adult education pro- 
grams offered through secondary institutions accounted for about 4 percent of the funds al- 


located. The remaining 96 percent of funds went to IHEs. 


Public two-year institutions (community colleges) were the largest proportion of postsec- 
ondary institution suberantees (83 percent), followed by public four-year institutions (10 
percent). These IHEs accounted for about 92 percent of all Title I postsecondary subgrant- 
ees. The remaining 8 percent of subgrantees were a mix of administrative units, public less- 


than-two-year institutions, and for-profit institutions, among others. More than four-fifths 


65 Sec. 3(22). 
66 Adult programs offer life-long learning opportunities in career and technical fields for adults in 
preparation for the workplace or postsecondary programs. 
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(87 percent) of all public two-year [HEs received a grant (Exhibit 2.27). In contrast, the pro- 
portion of public four-year IHEs among subgrantees was smaller, with about 20 percent of 


these types of institutions receiving grants. 


Exhibit 2.27. 
Number and percentage of public two- and four-year or above postsecondary institutions 
with Title I allocations: FY 2010 


Number Percent 
Institution type Number of IHEs receiving grants receiving grants 
Public two-year 958 831 87 
Public four-year and above 645 98 20 


Exhibit reads: In FY 2010, 831 of 958 public two-year institutions received a Perkins IV grant, accounting for 87 percent of these 
grantees in this category. 


NOTE: Public two- and four-year and above institutions accounted for 92.3 percent of Title I IHE subgrantees. Percentages are based on 
the 49 states for which FY 2010 data were available. 


SOURCE: Postsecondary State Director Survey Fiscal Data, 2009; Snyder and Dillow (2011). 


Consortium provision 
To qualify for an individual Perkins grant, an JHE must have a minimum funding eligibility 
of at least $50,000, according to the basic grant distribution formula. Those unable to 
achieve this funding threshold may either participate in a consortium in which funds are 
combined for joint projects that serve all members, or seek a waiver to allow the consortium 
to fund programs that don’t serve all members of the consortium. States have the option of 
granting a waiver to any IHE located in a rural, sparsely populated area, but only two states 


reported exercising this option in the 2008-09 program year, as compared to five states in 
2000-01.97 


In FY 2010, 10 of the 48 states (and the District of Columbia) for which local allocation data 
were available had from one to 26 postsecondary Title I consortia. Some 191 of the 1,197 
postsecondary subgrantees (about 16 percent) were members of one of 78 postsecondary 
consortia reported in FY 2010. The number of institutions participating by state ranged from 
two to 50, with the average consortia allocation varying from $56,000 to $375,000.° 


67 Secondary and Postsecondary State Director Surveys, 2009. 
68 Survey of State Directors of Career and Technical Education Fiscal Data, 2010. 
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Number and size of local allocations 
In FY 2010, 37 states allocated funds to 971 postsecondary Title I subgrantees. In FY 2001, 
the 38 reporting states reported a total of 1,065 allocations at the postsecondary level. The 
average allocation amount increased from $285,645 in FY 2001 to $320,496 in FY 2010, an 
increase of about 12 percent in nominal terms and a decline of about 9 percent in real 2010 
dollars (Exhibit 2.28). 


Exhibit 2.28. 
Number and average amount of Title I local postsecondary allocations in fiscal years 1992, 
2001, and 2010, and difference and percent change: FY 2001 through FY 2010 


Difference 
from 
FY 2001 

Number and through Percent 
amount of allocations FY 1992 FY 2001 FY 2010 FY 2010 change 
Number of allocations 996 1,065 971 -94 -9 
Average allocation size" 

Nominal dollars $226,019 $285,645 $320,496 $34,851 12 

Real dollars (2010) 351,282 351,703 320,496 -31,207 -9 


Exhibit reads: There were 1,065 Title I postsecondary allocations in FY 2001 and 971 in FY 2010, a decrease of 94 or about 9 percent. In 
nominal dollars, the average allocation in FY 2001 was $285,645 and in FY 2010, $320,496, an increase of $34,851 or about 12 percent. 


“Nominal dollars are the actual allocation amounts as they were allocated and reported in the given year. Real dollars have been adjusted 
for a measure of inflation. 

NOTE: Calculations do not reflect consortia memberships. Calculations for 1992 and 2001 are based on the 38 states for which data were 
available in both years;FY 2010 calculations exclude one additional state for which data were not available in that year. 

SOURCE: Postsecondary State Director Survey Fiscal Data, 2009; and Real dollar calculations: Bureau of Labor Statistics, Inflation 
Calculator, http://data.bls.zov/cgi-bin/cpicalc.pl (accessed January 21, 2011). 


The average allocation increased in 29 states and declined in 12 states between FY 2001 and 
FY 2010 (Exhibit 2.29). Average allocation amounts in FY 2010 ranged from $42,881 to 
$769,047, although one state reported allocating all postsecondary Title I funds ($1,080,743) 
to one subgrantee. In FY 2010, THE allocation amounts ranged from a maximum of 
$5,125,502 to a minimum of $223. 


© Agoregation of 2001 data did not allow the state for which 2010 data are not available to be exclud- 
ed in both years. 
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Exhibit 2.29. 
Average, maximum, and minimum postsecondary allocation amounts in FY 2001 and FY 
2010 and direction of average grant amount change, by state: FY 2001 through FY 2010 


Average allocation Maximum allocation Minimum allocation 
Change 
from 

FY 2001 

through 
State FY 2001 FY 2010 FY 2010 FY 2001 FY 2010 FY 2001 FY 2010 
National* $285,645 $320,496 + $4,832,430 $5,125,502 $1,025 $223 
Alabama 198,916 211,007 + 558,991 463,898 72,066 64,640 
Alaska 127,330 121,666 - 184,275 196,646 93,439 10,000 
Arizona 246,615 319,353 + 840,683 1,004,975 110,398 175,784 
Arkansas 115,288 118,477 + 223,805 393,889 52,202 16,667 
California 393,154 769,047 + 4,832,430 5,125,502 43,746 66,564 
Colorado 348,454 420,891 + 885,294 1,179,655 75,354 93,053 
Connecticut 123,711 133,454 + 184,175 242,490 72,783 34,241 
Delaware n/a 83,135 n/a n/a 92,700 n/a 71,908 
District of Columbia n/a 717,360 n/a n/a 717,360 n/a 717,360 
Florida 384,061 426,364 + 3,787,602 3,931,771 51,977 1,259 
Georgia 363,478 575,221 + 1,189,149 1,328,236 87,936 145,491 
Hawaii 2,285,140 320,106 n/a 2,285,140 397,198 2,285,140 249,809 
Idaho 316,451 329,284 + 572,433 616,268 162,861 211,768 
Illinois 336,405 390,888 + 3,642,996 3,686,543 79,165 108,699 
Indiana n/a 377,542 n/a n/a 1,019,371 n/a 21,538 
Iowa 299,652 334,225 + 692,196 834,924 77,311 76,174 
Kansas 151,331 155,075 + 405,891 339,003 51,690 25,435 
Kentucky 235,850 345,976 + 559,516 755,921 95,847 84,458 
Louisiana 167,455 n/a n/a 1,616,381 n/a 52,803 n/a 
Maine 1,973,707 339,104 n/a 1,973,707 611,727 1,973,707 154,418 
Maryland 271,075 322,412 + 1,015,893 1,323,289 42,197 23,434 
Massachusetts 266,556 193,744 - 492,892 533,471 64,079 3,003 
Michigan n/a 432,222 n/a n/a 1,594,215 n/a 60,687 
Minnesota 392,891 324,616 - 1,568,000 1,077,708 75,772 24,448 
Mississippi 396,133 357,062 - 1,098,349 820,694 142,643 175,618 
Missouri 290,978 124,204 - 1,316,712 1,298,762 60,579 307 
Montana 111,996 135,241 + 204,180 264,270 54,061 65,916 
Nebraska 400,833 414,920 + 747,358 946,970 152,803 36,168 
Nevada 373,857 526,730 + 846,572 1,228,771 104,866 142,806 
New Hampshire 193,044 1,080,743 + 291,782 1,080,743 30,669 1,080,743 
New Jersey 209,413 238,284 + 606,229 950,533 1,025 223 
New Mexico 274,703 225,902 = 1,464,811 1,296,213 63,448 50,235 


See notes at end of table. 
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Exhibit 2.29.—cont. 
Average, maximum, and minimum postsecondary allocation amounts in FY 2001 and FY 


2010 and direction of average grant amount change, by state: FY 2001 through FY 2010 


Average allocation 


Maximum allocation 


Minimum allocation 


67 


Change 
from 

FY 2001 

through 
State FY 2001 FY 2010 FY 2010 FY 2001 FY 2010 FY 2001 FY 2010 
New York $313,510 $354,799 + $1,472,275 $1,705,641 $55,852 $54,161 
North Carolina n/a 177,717 n/a n/a 728,659 n/a 53,741 
North Dakota 127,839 120,588 - 328,162 230,847 1,352 31,438 
Ohio 91,084 154,642 + 335,065 483,850 28,050 52,254 
Oklahoma 125,427 42,881 - 292,354 259,148 48,852 3,847 
Oregon n/a 345,464 n/a n/a 1,400,536 n/a 22,900 
Pennsylvania 222,648 162,069 - 1,620,096 1,005,561 44,363 1,774 
Rhode Island n/a 87,615 n/a n/a 350,919 n/a 50,000 
South Carolina 158,481 370,520 + 351,224 891,997 45,972 73,932 
South Dakota 393,251 325,659 - 669,646 671,416 23,713 91,528 
Tennessee 70,645 68,810 - 201,300 146,086 26,600 50,000 
Texas 538,625 526,601 - 2,199,000 2,470,525 54,000 50,000 
Utah 284,340 290,068 + 1,115,319 1,467,295 8,000 10,830 
Vermont 236,504 345,971 + 382,150 518,956 130,622 172,986 
Virginia 37,630 140,206 + 418,944 572,252 50,000 50,000 
Washington 297,578 360,535 + 1,360,944 1,258,209 65,310 32,873 
West Virginia 73,505 229,673 + 355,116 424,590 95,181 94,654 
Wisconsin 549,267 552,583 + 2,553,540 2,078,600 114,600 106,600 
Wyoming 177,378 216,077 + 279,014 414,355 97,757 122,665 
Exhibit reads: In FY 2010, the average Title I postsecondary local allocation in Alabama was $211,007, a increase (positive change) from 


the average allocation amount in FY 2001. The maximum allocation in Alabama that year was $463,898 and the minimum, $64,640. 


n/a Data missing or not available, or data are not comparable across years. 
+Average allocation size increased between FY 2001 and FY 2010. 


— Average allocation size decreased between FY 2001 and FY 2010. 
a The national average reflects the trend calculations in Exhibit 2.28. 


NOTE: Allotment amounts for FY 2010 are estimates based on formula calculations. Actual grants may vary according to consortia 


memberships. Amounts may also include carry over funds, which are unspent local funds from the previous year that are reallocated by 


formula. 


SOURCE: Silverberg et al. (2004); and Postsecondary State Director Survey Fiscal Data, 2009. 


Use of funding at the local level 


The required uses for Title I funds at the local level are the same for secondary and postsec- 


ondary education (see page 58).” The permissive uses apply to both levels as well, except for 


one new permissive use directed at the postsecondary level: developing and expanding post- 


79 Sec. 135(b). 
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secondary program offerings at times and in formats that are accessible for students, includ- 


ing working students, through the use of distance education.”! 


The most common permissive uses among postsecondary subgrantees in the 2008-09 pro- 
gram year were providing career guidance and academic counseling (81 percent), followed by 
leasing, purchasing, upgrading, or adapting equipment (76 percent) and providing programs 
for special populations (69 percent). Among the least common were creating small, personal- 
ized career-themed learning communities (11 percent), entrepreneurship education and train- 
ing (14 percent), and supporting family and consumer science programs (15 percent) 
(Exhibit 2.30). 


Case study interviews echoed the survey finding that career counseling and equipment ex- 
penses were the most common uses of Title I funds at the postsecondary level. Respondents 
noted that Perkins funds were used to support career advisor salaries and equipment up- 
grades for technology-dependent programs like healthcare. Among other uses noted by mul- 
tiple postsecondary subgrantees were expenditures for data specialists, staff professional 
development, and CTE program marketing. Although marketing was noted as a use of funds 
at both secondary and postsecondary levels, postsecondary respondents described more ex- 


tensive media campaigns that included advertising through print, radio, and other media. 


As at the secondary level, case study evidence suggests that postsecondary Perkins funds help 
keep programs up-to-date and meet industry standards. One local director explained, 
“[Those funds are critical] to support the extra lab hours..., the instructional techs in the 
labs to help the students get what they need, the equipment purchases.” Unlike at the sec- 
ondary level, however, none of the postsecondary faculty or staff indicated that this was the 
only source of funds available for these purposes. Several local directors described combin- 
ing Perkins funds with other sources, and two reported leveraging Perkins funds to attract 
funding from multiple academic units or outside sources, such as foundations, for large 


technology purchases such as computer lab upgrades and medical equipment. 


7 Sec. 135(c). 
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Percentage of postsecondary subgrant recipients who reported using Title I funds for 


Exhibit 2.30. 


permissive uses: Program year 2008-09 


Permissive use 

Providing career guidance and academic counseling 

Leasing, purchasing, upgrading, or adapting equipment 

Providing programs for special populations 

Implementing CTE programs of study (POS) 

Supporting nontraditional training and activities 

Promoting work-related experiences for students 

Offering mentoring and related support services 

Developing new CTE courses 

Involving business and labor in designing, implementing, 
and evaluating CTE programs covered by the Act 

Promoting industry experiences for teachers 

Providing training programs in automotive technologies 

Improving accountability data collection and reporting 

Supporting teacher preparation programs 

Offering continuing education or job referral services 

Offering programs for adults and school dropouts 

Assisting CTE student organizations 

Supporting family and consumer sciences programs 


Entrepreneurship education and training 


Creating small, personalized career-themed learning communities 


Other* 


Percentage 
of recipients 


81 
76 
69 
61 
61 
57 
48 
46 


45 
40 
40 
36 
32 
27 
26 
23 
15 
14 
11 

6 
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Exhibit reads: In program year 2008-09, 81 percent of postsecondary grant recipients reported using Perkins funds for providing career 


guidance and academic counseling. 


““Other” includes faculty training and professional development, career academies, integrating academics and CTE, technology, salaties, 
supplies, materials, five-year planning, career software, student credentials and industry exams, and inquiry-based learning. 


NOTE: Estimates were weighted to reflect population values. 


SOURCE: THE Survey, 2009. 
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Title Il (Tech Prep) 


Perkins I introduced Title 1 programs as part of the special projects section of the legisla- 
tion (Title IIE), and Congress successively reauthorized the program in 1998 and 2006.” 
Although Title Il was retained in Perkins IV, Congress eliminated funding for this program 
in FY 2011. Beginning in FY 2012, states no longer received Title II resources. Before the 
funds for the program were discontinued, Perkins IV gave states the option to consolidate 
their Title Hf funds into their Title I basic grant. States that merged funds were exempt from 
reporting on the nine new accountability provisions and were no longer required to dis- 


ageregate Title I program students on the secondary and postsecondary core measures. 


The education programs that Title II resources supported are generally known as Tech Prep 
programs. These programs are carried out under a formal articulation agreement between 
secondary and postsecondary subgrantees participating in a Tech Prep consortium.7> A Tech 
Prep program combines at least two yeats of secondary education with at least two years of 
postsecondary education, with coursework offered in a coherent, nonduplicative sequence. 
Programs integrate academic and CTE instruction, offer work-based learning opportunities 
where appropriate and available, and must lead to technical skill proficiency, an industry- 
recognized credential, or a degree or certificate in a specific career field. Programs also 
should prepare students for employment in a high-skill, high-wage, or high-demand occupa- 


tion or for further postsecondary education. 


Title Il (Tech Prep) Appropriations and Grants to States 


In nominal terms, federal support for Title II rose by about $43 million (67 percent) between 
FY 1991 (Perkins I) and FY 1999 (Perkins ID), but it declined from $106 million in FY 1999 
(Perkins II) to about $105 million in FY 2007 (Perkins IV) (Exhibit 2.31). In real 2010 dol- 
lars, the purchasing power of Title IT funds fell by about $28 million from FY 1999 to FY 
2007, a decline of about 20 percent. 


72 Sec. 201(a). 
7 Sec. 203(c). 
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Exhibit 2.31. 
Federal Perkins Title II appropriations in nominal and real 2010 dollars (in millions of 
dollars): Fiscal years 1991 (Perkins IT), 1999 (Perkins II), and 2007 (Perkins IV) 


Federal 
approptiations 
$160 

140 

120 

100 

80 


Perkins IT Perkins LIT Perkins IV 
FY 1991 FY 1999 FY 2007 


B Nominal dollars @ Real 2010 dollars 


71 


Exhibit reads: In nominal dollars, the federal appropriation for Perkins Tech Prep (Title H) programs was about $106 million in FY 1999 


and $105 million in FY 2007. In real 2010 dollars, the federal appropriation was $110 million in FY 2007 and $139 million in FY 1999. 
SOURCE: State Funding History Tables, FY 1980 to FY 2008, http://www2.ed.gov/about/overview/budget/history/index.html 
(accessed June 12, 2010); FY 2010 (to date): U.S. Department of Education Fiscal Year 2011 President's Request, 
http://www2.ed.gov/about/overview/budget/budget11/summary/appendix4.pdf (accessed June 12, 2010); and Real dollar calculations: 
Bureau of Labor Statistics, Inflation Calculator, http://data.bls.gov/cei-bin/cpicalc.pl (accessed January 21, 2011). 


The formula used to distribute Title I funds also determined the amount of a separate annual 
Title II allocation to all states. States seeking funds were required to submit an application to 
the Department as part of their state plan, describing how such programs would coordinate 
with other state activities. States opting to merge Title I and II funding had to notify the 
government of their consolidation plans, after which they could manage Title I funds as 
they would those allocated under Title I. Title II grants to states in FY 2009 ranged from a 
maximum of about $11 million to a minimum of $54,653 (Exhibit 2.32). 


Reflecting the decline in federal Title H appropriations, grants to all states declined between 
FY 2001 and FY 2009. Some 19 of the 53 state grantees experienced losses of just one-tenth 
of 1 percent, but the grants declined by more than 5 percent for 15 state grantees. 
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Exhibit 2.32. 


72 


Federal Title II allocation amounts to states, fiscal years 1992, 2001, and 2009, 
and percent change, by state: FY 2001 through FY 2009 


State 
Alabama 
Alaska 
Arizona 
Arkansas 
California 
Colorado 
Connecticut 


Delaware 


District of Columbia 


Florida 
Georgia 
Hawaii 

Idaho 

Illinois 
Indiana 

lowa 

Kansas 
Kentucky 
Louisiana 
Maine 
Maryland 
Massachusetts 
Michigan 
Minnesota 
Mississippi 
Missouri 
Montana 
Nebraska 
Nevada 

New Hampshire 
New Jetsey 
New Mexico 
New York 
North Carolina 
North Dakota 
Ohio 


See notes at end of table. 


FY 1992 
$1,808,501 
160,609 
1,448,578 
1,004,895 
9,357,339 
1,119,563 
880,622 
217,099 
179,938 
4,063,397 
2,616,728 
358,614 
435,623 
3,665,308 
2,228,749 
1,042,639 
874,276 
1,626,968 
1,879,665 
459,176 
1,343,891 
1,704,461 
3,395,318 
1,475,885 
1,209,134 
1,863,805 
317,518 
578,707 
379,529 
328,238 
2,068,361 
644,953 
5,153,396 
2,785,654 
269,429 
4,069,951 


= 


FY 2001 


$1,995,785 


336,753 
1,987,848 
1,186,934 
895,013 
420,073 
869,581 
440,480 
321,892 
5,075,903 
3,102,535 

530,000 

652,082 
4,054,595 
2,465,494 
1,245,235 
1,120,100 
1,863,661 
2,190,094 

530,000 
1,575,660 
1,658,556 
3,768,685 
1,736,576 
1,390,909 
2,217,159 

519,374 

709,518 

576,717 

530,000 
2,192,627 

843,258 
5,246,770 
2,995,591 

376,267 
4,449,520 


FY 2009 
$1,994,293 
250,163 
1,880,272 
1,186,047 
11,251,825 
1,393,615 
868,931 
229,550 
134,677 
4,815,873 
3,074,414 
411,510 
623,977 
4,049,330 
2,463,651 
1,244,304 
1,065,569 
1,862,269 
2,188,457 
525,512 
1,540,547 
1,648,213 
3,641,767 
1,735,278 
1,389,869 
2,177,836 
428,023 
708,988 
527,487 
376,159 
2,187,235 
833,848 
5,242,848 
2,993,352 
313,151 
4,446,194 


Percent change from 
FY 2001 through FY 2009 
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Exhibit 2.32.—cont. 
Federal Title II allocation amounts to states, fiscal years 1992, 2001, and 2009, 
and percent change, by state: FY 2001 through FY 2009 


Percent change from 


State FY 1992 FY 2001 FY 2009 FY 2001 through FY 2009 
Oklahoma $1,307,304 $1,587,998 $1,569,862 -1 
Oregon 1,014,963 1,299,575 1,290,703 -1 
Pennsylvania 4,071,449 4,238,522 4,235,353 0 
Rhode Island 354,695 530,000 338,543 -36 
South Carolina 1,593,356 1,738,505 1,730,085 -1 
South Dakota 282,514 426,704 352,942 -17 
‘Tennessee 2,037,127 2,233,311 2,231,641 0 
Texas 6,992,837 8,495,203 8,391,458 -1 
Utah 818,698 1,226,873 1,195,557 3 
Vermont 214,951 352,887 237,188 -33 
Virginia 2,104,071 2,445,828 2,417,795 -1 
Washington 1,628,578 2,091,644 2,036,850 3 
West Virginia 783,267 877,270 876,614 0 
Wisconsin 1,805,496 2,128,276 2,105,227 -1 
Wyoming 182,117 280,263 233,729 -17 
Puerto Rico 1,742,268 1,921,202 1,919,766 

Virgin Islands 49,792 54,694 54,653 0 
Total $90,000,000 $106,000,000 $102,923,000 -3 
Exhibit reads: In FY 2001, the Tech Prep allocation amount for Alabama was $1,995,785, and $1,994,293 in FY 2009, a decline of .1 
percent. 


SOURCE: U.S. Department of Education, Office of the Deputy Secretary, Budget Service. State Funding History Tables by Program, 
http://www.ed.gov/about/overview/budget/history/index.html?src=rt (accessed June 16, 2010); and 
http://www2.ed.gov/about/ovetview/budget/statetables/11stbyprogram.pdf (2009 data, accessed February 9, 2011). 


Consolidation of Title | and Il funds 
Until FY 2012 under Perkins IV, states could choose whether to consolidate their Title I and 
II funds each year. During the first year of Perkins IV, several states maintained separate 
funding for Title IT and then merged funds in later years, and the fiscal and case study data 


did not suggest that any state ever reversed the decision. 


As of FY 2010, a total of 27 states (including Puerto Rico) opted to merge their Title I 
funds with their Title I allocation. * Only one state, Alabama, opted for a partial consolida- 


74 States that consolidated Title I and II funds are shown in Exhibit 2.35, which is discussed in a later 
section. 
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tion and merged 60 percent of its funding with its Title I grant, with remaining funds dis- 


tributed among Tech Prep consortia on a competitive basis. 


Secondary and postsecondary state directors provided a variety of reasons for their decisions 
to consolidate Title I and H funds (see Chapter 3, Exhibit 3.31). The most common reason 
at the secondary level, chosen by 23 of the 27 secondary directors responding, was the desire 
to incorporate Tech Prep education programs into all CTE programs, followed by the simi- 
larity between Tech Prep and programs of study (18) and to avoid the data burden associated 
with collecting new Title II measures (15). Responses from postsecondary directors were 
similar, although fewer cited a desire to incorporate Tech Prep into all CTE programs (18 
out of 27 responses), which tied with data burden as the most common response. Case study 
evidence indicated that concerns about consolidating funds included the possible loss of 
funds to the secondary or postsecondary level and the sustainability of existing Tech Prep 


programs if dedicated funding were discontinued. 


The state director survey also asked directors from states that had not consolidated funds 
whether they planned to consolidate during the next two years. Of the 24 responses from 
secondary state directors, none reported plans to consolidate by 2011—12.75 Case study evi- 
dence indicates that consolidation decisions involve administrators beyond those directly in- 


volved with CTE, including chancellor’s offices and state boards of education. 


Title Il local allocations 
As noted earlier, states maintaining separate Title H funds award grants to Tech Prep con- 
sortia. Consortia must consist of a secondary LEA, intermediate education agency, atea CTE 
center, BIA-funded school, and a nonprofit or proprietary postsecondary institution offering 
a two-year associate’s degree or apprenticeship program.’ Grants may be distributed com- 


petitively or according to a formula determined by the state agency. 


Among the states or territories that had maintained separate funding as of program year 
2008-09, nine states allocated by formula, nine states through a competitive grant process, 
and another five states used both or other approaches (Exhibit 2.33). State formulas vary 
from simple to complex. For example, South Dakota allocated its Title II funds equally 
across each of its four consortia (South Dakota Board of Education 2008), while Missouri 
used a three-part formula that provided a base amount of $105,000 to each of the 12 consor- 
tia in the state, with remaining funds split 40 percent based on the percentage of Tech Prep 
enrollment (maximum of $50,000 per consortium) and 60 percent based on each consorti- 
um’s pro rata share of statewide enrollment (Missouri Department of Elementary and Sec- 
ondary Education 2008). 


% 'The secondary-level responses are highlighted because only 19 postsecondary directors responded. 
76 Sec. 204(a)(1). 
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Exhibit 2.33. 
Among states allocating Title II (Tech Prep) funds, number using various methods to 
award consottia grants: Program years 2000-01 and 2008-09 


Number of states 


Method used 2000-01* 2008-09” 
By formula 21 9 
Through a competitive process 14 9 
Through a combination of formula and competition 6 4 
Other il 
No response 0 2 


Exhibit reads: In 2000-01, 21 states allocated their Tech Prep funds by using a formula. 
“N=46 states. 


> N=25 states. 
SOURCE: Silverberg et al. (2004); and Secondary and Postsecondary State Director Surveys, 2009 


As a result of states’ consolidation of their Title I and II funds, the number of allocations to 
Tech Prep consortia reported declined by about half (53 percent) between FY 2001 and FY 
2010, falling from 738 to 349 (Exhibit 2.34). In real 2010 dollars, the average allocation 
amount declined by about $7,000 from FY 2001 to FY 2010, changing from $136,779 to 


$144,031. 
Exhibit 2.34. 
Number and average amounts of Title II consortia allocations: Fiscal years 1994, 1996, 2001, 
and 2010 
FY 1994 FY 1996 FY 2001 FY 2010 
Number of grants awarded 953 1,029 738 349 
Average consottium grant amount" 
Nominal dollars $117,273 $100,148 $111,089 $144,031 
Real dollars (2010) 172,551 139,183 136,779 144,031 


Exhibit reads: In FY 2001, 738 Title II grants were awarded. The average grant amount was $111,089 in nominal dollars and $136,779 in 
2010 real dollars. 

“Nominal dollars are the actual grant amounts as they were allocated and reported in the given year. Real dollars have been adjusted for a 
measure of inflation. 

FY 1994 and FY 1996: N=50 states. 

FY 2001: N=40 states. 

FY 2010: N=23 of the 25 states that maintained separate Title I and Title Hl funds in 2009-10. 

SOURCE: Silverberg et al. (2004); and Secondary and Postsecondary State Director Surveys Fiscal Data, 2009. Real dollar calculations: 
Bureau of Labor Statistics, Inflation Calculator, http://data.bls.gov/cgi-bin/cpicalc.pl (accessed January 21, 2011). 


Allocation amounts ranged from a maximum of $986,636 to a minimum of $16,667 in FY 
2010. Among the 21 states for which the average allocation amount is available in both years, 


the amount increased in 13 states and declined in 8 (Exhibit 2.35). 
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Exhibit 2.35. 
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Average Title II local allocation amounts and direction of average amount change, FY 2001 


through FY 2010, and maximum and minimum allocations awarded, by state: FY 2010 


State 


National* 


Alabama” 
Alaska 
Arizona 
Arkansas 
California 
Colorado 
Connecticut 
Delaware 
District of Columbia 
Florida 
Georgia 
Hawaii 
Idaho 
Illinois 
Indiana 
Iowa 
Kansas 
Kentucky 
Louisiana 
Maine 
Maryland 
Massachusetts 
Michigan 
Minnesota 
Mississippi 


Missouri 


Montana 


See notes at end of table. 


Average allocation 


FY 2001 
$111,089 


77,193 
118,250 
112,901 

96,219 
117,647 

27,524 

73,727 

n/a 
n/a 
172,218 
n/a 
530,000 
103,395 
91,140 
n/a 
66,413 
n/a 

37,393 
149,213 

n/a 

64,368 

131,302 
n/a 

53,200 
123,906 
190,258 
102,880 


FY 2010 
$144,031 


50,000 
266,786 
168,351 
‘ 
324,125 
: 
‘ 


2K 


101,300 


eK 
79,292 
rt 

£ 

% 

% 


* 


117,170 
145,825 

£ 
102,936 
146,710 
428,023 


Change from 


FY 2001 
through 
FY 2010 


+ 


FY 2010 
Maximum Minimum 
$968,636 $45,000 
50,000 50,000 
266,786 266,786 
292,000 112,000 
n/a n/a 
580,001 150,000 
n/a n/a 
n/a n/a 
n/a n/a 
n/a n/a 
n/a n/a 
n/a n/a 
n/a n/a 
n/a n/a 
495,168 47,362 
n/a n/a 
115,724 65,822 
n/a n/a 
n/a n/a 
n/a n/a 
n/a n/a 
n/a n/a 
186,338 64,764 
348,066 49,694 
n/a n/a 
102,936 102,936 
265,611 114,309 
428,023 428,023 
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Exhibit 2.35.—cont. 
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Average Title II local allocation amounts and direction of average amount change, FY 2001 
through FY 2010, and maximum and minimum allocations awarded, by state: FY 2010 


State 

Nebraska 
Nevada 

New Hampshire 
New Jetsey 
New Mexico 
New York 
North Carolina 
North Dakota 
Ohio 
Oklahoma 


Oregon 
Pennsylvania 
Rhode Island 
South Carolina 
South Dakota 
Tennessee 
Texas 

Utah 
Vermont 
Virginia 
Washington 
West Virginia 
Wisconsin 


Wyoming 


n/a 
$138,054 
100,000 
99,813 
118,164 
191,442 
n/a 
$55,153 
156,548 
69,600 
n/a 
255,860 
n/a 
172,739 
94,715 
115,007 
297,522 
134,479 
56,410 
89,652 
86,466 
97,313 
127,899 
27,159 


Average allocation 


FY 2001 FY 2010 


$112,000 
* 
396,078 
390,496 
87,077 

* 

176,301 
51,429 


* 
OK 
* 
* 


76,986 


* 
319,722 
* 


* 


102,079 
91,733 
92,333 

124,985 


* 


Change from 
FY 2001 
through 
FY 2010 


n/a 


n/a 


Maximum 


n/a 
n/a 
$112,000 
n/a 
443,608 
937,475 
197,000 
n/a 
365,708 
65,000 
n/a 
n/a 
n/a 
n/a 
76,986 
n/a 
968,636 
n/a 
n/a 
191,658 
141,409 
100,000 
173,528 


n/a 


Minimum 


n/a 
n/a 
$112,000 
n/a 
348,548 
92,062 
50,000 
n/a 
114,231 
45,000 
n/a 


n/a 


76,986 
n/a 
218,013 
n/a 
n/a 
95,829 
74,166 
84,000 
88,632 


n/a 


Exhibit reads: The average Title II local allocation amount was $111,089 in FY 2001 and $144,031 in FY 2010, an increase (positive 
change) from FY 2001 to FY 2010. The maximum and minimum allocation amounts awarded in FY 2010 were $968,636 and $45,000, 


respectively. 


* Consolidated funds by FY 2010 (Source: U.S. Department of Education). 


** Did not consolidate funds by program year 2008-09, but Tech Prep allocation data at the consortia level are not available. 
n/a Data missing or not applicable, or data are not comparable across years. 


+Average allocation amount increased between FY 2001 and FY 2010. 
— Average allocation amount decreased between FY 2001 and FY 2010. 


“The national average allocation is the average of the average allocation for all states for which data are available in each year. 


> Alabama partially consolidated and merged 70 percent of Title II funds with Title 1 funds. 
SOURCE: Silverberg et al. (2004); and Secondary and Postsecondary State Director Surveys Fiscal Data, 2009. 
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Case study information indicated that consortia used Title II funds for several purposes re- 
lated to Tech Prep programs. The funds were used for personnel to assist with articulation 
agreement negotiations and to support meetings between secondary and postsecondary ad- 
ministrators and faculty. Local administrators also said that some funds were used for pro- 
gram websites and for the compilation of student data for federal reporting purposes. 
Several local administrators also emphasized the similarities between Tech Prep programs 
and POS, noting that some Tech Prep funds were being used to support the development 


and implementation of POS. 
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Chapter 3. Accountability 


Building upon accountability requirements introduced in earlier Perkins legislation, the Carl 
D. Perkins Career and Technical Education Act of 2006 (Perkins IV) continues to hold states and 
local career and technical education (CTE) providers responsible for collecting and reporting 
data about the performance of the Perkins grant program. As in prior legislation, Perkins IV 
offers states considerable flexibility in determining how to collect and report accountability 
data but asks for improved data quality in return. The U.S. Department of Education (the 
Department) is supporting this higher standard by offering guidance to assist state and local 


efforts to report valid, reliable, and comparable performance data. 
The analyses in this chapter seek to answer the following research questions: 


e How are states and local subgrantees implementing Perkins IV accountability 


requirements? 


e What challenges are states and local subgrantees facing, and how are these chal- 


lenges being resolved? 


e Are states and local subgrantees setting and meeting performance targets, and 


what is happening when targets are not met? 
e How are data used to promote program improvement and student success? 


This chapter describes the evolution of accountability under prior Perkins legislation; changes 
introduced in Perkins IV’; and progress by the Department, states, and local subgrantees in 
designing and implementing the new accountability framework. It also explores efforts to in- 
crease the validity, reliability, and comparability of reported data, and the challenges federal, 
state, and local staff face in ensuring the accuracy of performance data. Finally, it investigates 
how states and local subgrantees are setting performance goals, meeting performance targets, 


and using data to improve programs and student outcomes. 


This chapter draws on information from multiple qualitative and quantitative sources. Re- 
sults of the state director, local education agency (LEA), and institution of higher education 
(HE) surveys offered insights into how states and local providers ensure data validity and 
reliability, set and meet performance goals, and use performance results to improve pro- 
grams and student outcomes. Case studies in six states and 18 school districts and institu- 
tions provided in-depth information regarding these issues. Extant data sources supported 


the original data collected for the study, and included relevant literature as well as the De- 
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Key 


partment’s nonregulatory guidance; annual reports submitted to Congress regarding Perkins 
performance; and findings from Department-sponsored technical assistance activities, such 
as Data Quality Institutes (DQIs) and customized technical assistance for individual states.! 
Each source contributed to a greater understanding of the accountability framework and 
how states and local subgrantees are implementing the requirements. More information 


about these data sources and the study methodology is available in Appendix A. 


Findings 


The folowing observations emerged from this analysis of states’ implementation of Perkins 


accountability provisions and are discussed in greater depth in this chapter. 


1. Most states and local subgrantees are working to improve the quality of their Perkins IV accountability 


SYStCMS. 


States and local providers have continued to develop and refine the systems they use to col- 
lect and report Perkins performance data. Most states (49 secondary, 44 postsecondary) have 
adopted more than one strategy to improve the quality, validity, and reliability of locally re- 
ported data. Common strategies included providing technical assistance or guidance on col- 
lecting and editing data, performing electronic error checking, and conducting desk audits of 
local data. LEAs and IHEs rely primarily on state measure guidance to promote validity and 
reliability. States and local subgrantees also continued to refine the data systems they use to 
collect and report data through initiatives such as introducing common student identifiers 
across education levels and connecting Perkins data systems with other education data sys- 
tems. The Department is supporting these efforts by offering multiple avenues to obtain 
technical assistance, as well as providing nonregulatory guidance designed to promote valid, 


reliable, and comparable reporting across states. 


2. Perkins IV and the Department’s nonregulatory guidance offer states flexibility in how they interpret 
accountability requirements, contributing to variation in data collection and reporting practices across 


states. 


Changes to the accountability framework in Perkins [V—such as the addition of separate in- 
dicators for postsecondary—coupled with the Department’s guidance have refined expecta- 
tions for data collection and reporting. The guidance and legislation both offer states 
flexibility in determining what indicators to use and how to define population and perfor- 
mance measures. Although many states at least consulted the nonregulatory guidance for one 


or more population and measure definitions, only nine states used the nonregulatory guid- 


! Many of these sources and documents are available on the Department-sponsored Perkins Collabora- 
tive Resource Network website (http://cte.ed.gov). 
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ance verbatim when defining all of their secondary populations and performance measures, 
and only five states did the same for every postsecondary population and performance 
measure. The flexibility inherent in the legislation and nonregulatory guidance may reduce 


the Department’s ability to aggregate data at the national level. 


3. Capacity and infrastructure influence how states and local subgrantees approach data collection and report- 


ng. 


States and local subgrantees offer CTE programs within different governance, resource, and 
infrastructure systems. Issues such as access to necessary data, timing, cost, capacity of stu- 
dent information systems, and legal interpretations of federal privacy laws continue to affect 
the ability of some states and local providers to implement Perkins accountability require- 
ments. The postsecondary placement indicator is an example of the challenges some states 
may face: while providers are required to report whether students are employed, in the mili- 
tary, ot enrolled in an apprenticeship program after leaving their institution, a few states did 
not have the legal authority to access state employment databases to obtain students’ labor 
market outcomes. These states may have relied upon surveys or interviews to gather stu- 
dents’ self-reported labor market outcomes. More than 30 postsecondary state directors in- 
dicated the cost of matching student records with other state or national databases had at 
least some impact on their ability to report results for this indicator. These and other differ- 
ences contribute to variations in how states report Perkins IV performance results and may 


result in less data comparability. 


4. States have implemented performance requirements for locals, although the effect of these requirements has 
yet to be fully measured. 


Many state directors reported that their states had existing policies that required subgrantees 
to implement a local improvement plan if they failed to meet targets on one or more core 
indicators (47 secondary, 38 postsecondary). Several states also had policies for restricting or 
eliminating local subgrantees’ flexibility in spending Perkins funds (30 secondary, 22 postsec- 
ondary), although very few had policies to reduce or eliminate local subgrantee funding (11 
secondary, 9 postsecondary). Sixty-five percent of LEA directors and 78 percent of IHE di- 
rectors reported that their state would require a program improvement plan if a local sub- 
grantee failed to meet one or more targets. About half of local directors indicated that 
negotiating local targets would have no impact on their administration and implementation 
of CTE programs, and about one-quarter thought it would have a somewhat to very positive 
impact. Among those LEAs and IHEs that negotiated performance targets, most experi- 
enced little to no difficulty during their negotiations with the state. More than half of local 
directors (61 percent of LEAs and 67 percent of IHEs) also reported that introducing sanc- 


tions would have no impact on local CTE administration and implementation, while very 
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few (15 percent of LEA and 11 percent of IHE directors) said it would have a somewhat to 


very positive impact.” 


5. States and subgrantees use data to identify programs in need of improvement and provide technical assis- 


tance. 


States, LEAs, and IHEs have made efforts to use data to improve program and student out- 
comes. More than half of states provided local subgrantees with statewide averages of per- 
formance results (40 secondary, 29 postsecondary), and many shared performance results for 
individual LEAs and IHEs (35 secondary, 30 postsecondary). Most state directors reported 
using Perkins accountability data to identify programs in need of improvement. While most 
states also used Perkins data to provide targeted technical assistance (45 secondary, 40 post- 
secondary), only about one-third of local subgrantees indicated that their state had a policy 
of providing professional development for staff in underperforming programs. About half of 
local subgrantees used Perkins data quite a bit or to a great extent to identify programs that 
need improvement (49 percent of LEAs and 47 percent of IHEs) or make program funding 
decisions (49 percent of LEAs and 48 percent of IHEs). 


The Accountability Framework 


The Carl D. Perkins Vocational and Applied Technology Education Act of 1990 (Perkins Il) laid the 
foundation for the accountability framework in use today. Perkins IT required states to devel- 
op a system of performance standards and measures of learning and competency gains— 
including attainment of basic and advanced academic skills—and at least one measure of 
performance, such as job or skill attainment, retention in or completion of secondary school, 
ot placement into employment or further training. Statewide systems also had to encourage 
setvices for targeted populations and outline procedures for using existing methods and re- 


sources developed under other federal programs (Perkins LI Sec. 115). 


Perkins II allowed states to choose which competency indicators and performance measures 
to incorporate into their accountability systems, without any requirement that the selected 
measures or methods be consistent across states. The legislation also did not require states to 
submit the results of their measures to the Department or any other federal agency. Perkins I 
called for the secretary of education to submit a report within four years that included a de- 


scription of the status of each state’s system of standards and measutes; an assessment of the 


? The surveys did not provide further insight into local subgrantees’ reasons for reporting that sanc- 
tions would have no effect or a positive effect. It is possible, however, that the timing of the survey 
played a part. Local directors responded to the survey in late 2009, at the same time they were submit- 
ting the first full year of Perkins IV data. They had not yet had extensive experience with new state 
policies regarding sanctions for failing to meet negotiated targets. 
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validity, reliability, and predictive capacity of the measures; and an evaluation of how compa- 


rable the standards and measures were across states. 


The Carl D. Perkins Vocational and Apphed Technology Education Act of 1998 (Perkins IID) built 
upon Perkins I by outlining a single set of four outcome indicators for both secondary and 
postsecondary students. The legislation required states to set performance targets for each 
indicator and report progress in meeting those targets for all students, including special pop- 
ulations.> Perkins II permitted states to develop their own methods for reporting results and 
offered them the option of reporting on existing state performance indicators as long as they 


were aligned with those in the legislation. According to the Department: 


[While this statutory flexibility has enabled states to meet their individual 
CTE program needs, it has produced dramatic inconsistencies among states 
in how student populations are defined and has enabled many states to use 
less valid and reliable measures, particularly for academic attainment and 
technical skills attainment. As a result, the Department has been unable to 
make comparisons of student performance across states or track the perfor- 


mance of students over time (U.S. Department of Education 2010c, p. 9). 


New Provisions in Perkins IV 


Departing from earlier legislation, Perkins IV outlines separate sets of core indicators for 
secondary and postsecondary students, modifies the list of special populations, and intro- 
duces nine performance measures for Tech Prep (Exhibit 3.1). The legislation also incorpo- 
tates secondary accountability indicators from the Ezmentary and Secondary Education Act 
(ESEA).* Perkins IV requires data and indicators to be valid and reliable and extends the ex- 
pectations for reporting and continuous improvement to the local level. LEAs and IHEs 
must now negotiate performance targets for each indicator with the state and are responsible 


for meeting those targets. 


3 Perkins III special populations included individuals with disabilities; individuals from economically 
disadvantaged families, including foster children; individuals preparing for nontraditional training and 
employment; single parents, including single pregnant women; displaced homemakers; and individuals 
with other barriers to educational achievement, including individuals with limited English proficiency 
(Sec. 3(23)). 

4 Although Perkins IV introduces the programs of study (POS) concept, it does not include any indica- 
tors or specific accountability requirements related to POS. 
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Exhibit 3.1. 


84 


Core indicators for Perkins I]IT and Perkins IV 


Perkins III 


Perkins IV 


Secondary and Postsecondary 


Secondary 


Postsecondary 


Academic and technical skill attain- 
ment: Student attainment of challenging 
State established academic, and vocational 
and technical, skill proficiencies. 


Academic attainment: Student attainment 


of challenging academic content standards 
and student academic achievement stand- 
ards, as adopted by the State in for ESEA 
and measured by the State determined 


proficient levels on the academic assess- 
ments under ESEA. 


Technical skill attainment: Student at- 
tainment of career and technical skill profi- 
ciencies, including student achievement on 
technical assessments that are aligned with 
industry-recognized standards, if available 
and appropriate. 


Technical skill attainment: Student at- 
tainment of challenging career and tech- 
nical skill proficiencies, including student 
achievement on technical assessments that 
are aligned with industry-recognized stand- 
ards, if available and appropriate. 


Completion: Student attainment of a 
secondary school diploma or its recog- 
nized equivalent, a proficiency credential 
in conjunction with a diploma, or a post- 
secondary degree or credential. 


Completion: Student rates of attainment 
of secondary school diploma; General Ed- 
ucational Development (GED); and profi- 
ciency credential, certificate, or degree, in 
conjunction with a secondary school di- 
ploma (if such credential, certificate, or 
degree is offered by the State in conjunc- 
tion with a secondary school diploma). 


Completion: Student attainment of an 
industry-recognized credential, a certificate, 
or a degree. 


Graduation rate: Student graduation rates 
as described in ESEA. 


Placement and retention: Placement in, 
retention in, and completion of, postsec- 
ondary education or advanced training, 
placement in military service, or place- 
ment or retention in employment. 


Placement: Student placement in postsec- 
ondary education or advanced training, in 
military service, or in employment. 


Retention or transfer: Student retention in 
postsecondary education or transfer to a 
baccalaureate degree program. 


Placement: Student placement in military 
service of apprenticeship programs or 
placement or retention in employment, 
including placement in high skill, high 
wage, ot high demand occupations or pro- 
fessions. 


Nontraditional participation and com- 
pletion: Student participation in and 
completion of vocational and technical 
education programs that lead to nontradi- 
tional training and employment. 


Nontraditional participation and com- 
pletion: Student participation in and com- 
pletion of career and technical education 


programs that lead to non-traditional fields. 


Nontraditional participation and com- 
pletion: Student participation in, and com- 
pletion of, career and technical education 
programs that lead to employment in non- 
traditional fields. 


Exhibit reads: Perkins IV outlines a separate set of core indicators for secondary and postsecondary students and incorporates 
secondary accountability measures from ESEA. 
SOURCE: The Carl D. Perkins Vocational and Applied Technology Education Act of 1998 and the Carl D. Perkins Career and Technical 


Education Act of 2006. 
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To adjust to these new requirements, states had the option of submitting a one-year transi- 
tion plan for 2007—08, the first program year covered by Perkins IV. For the transition year, 
the Department required states to submit results only for the secondary ES'FEA-aligned indi- 
cators: academic attainment and graduation rate (Justesen 2007b). States already had estab- 
lished reporting processes for the ESEA indicators and therefore only had to develop a 
secondary concentrator definition to allow them to identify the subset of all students to in- 


clude in the indicator results for Perkins reporting.5 


The Department asked states to collaborate with their local CTE communities during the 
transition year to develop and submit plans for the remaining five program years of Per- 
kins IV. States’ five-year plans included measure definitions for every population and indica- 
tor, and states began reporting results for all secondary and postsecondary core indicators in 
2008-09. States must update their five-year plans annually and may request approval for 
changes to indicator definitions and performance targets at that time. States set future per- 


formance targets in the first, third, and fifth years of Perkins IV. 


Validity and Reliability 


A report from the Office of Management and Budget (OMB) identified a significant flaw in 
Perkins II accountability: the lack of comparable performance information across states (Of- 
fice of Management and Budget n.d.). OMB rated the Perkins III program “Ineffective,” not- 
ing that the “validity and reliability of program performance data are limited” (Office of 
Management and Budget n.d.).’ Perkins III offered states the flexibility to define their own 
methodology for collecting and reporting indicator data, which resulted, according to OMB, 
in a vatiety of measurement approaches that “do not shate a common standard for validity 


and reliability and do not allow for ageregation.”§ 


References to valid and reliable data and indicators in Perkins IV emphasize the importance of 
data quality and set a higher expectation for performance reporting. The legislation does not 


specify, however, how indicators should be defined and reported, nor does it specify stand- 


> “Concentrators” are CTE students who meet a state-defined threshold for CTE coursetaking. Con- 
centrators represent the cohort of students considered in most of the performance measures under 
Perkins IV. For more details, refer to the discussion of concentrators in the section Student Popula- 
tions (p. 103). 

6 See the Performance section (p. 148) for a detailed discussion of performance targets. 

7 OMB used the Program Assessment Rating Tool (PART) to assess Perkins. PART uses a standard 
set of 25 questions relating to program performance and management. More information about 
PART is available at www.ExpectMore.gov. 

8 http://www.whitehouse.gov/omb/expectmore/summary/10000212.2002.html (May 28, 2010). 
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ards for validity and reliability. As a result, Perkins IV does not fully address a core challenge 
for Perkins accountability: the flexibility afforded states in defining their own indicators may 
create challenges in aggregating data at the national level (U.S. Government Accountability 
Office 2009; U.S. Department of Education 2010c). States also may use different standards 
for validity and reliability, as noted by OMB (n.d.) in its assessment of Perkins III. Despite 
these challenges to obtaining comparable data, the Department, states, and local subgrantees 


employ a variety of strategies to promote data quality, validity, and reliability. 


Department data quality initiatives 
Following the reauthorization of Perkins IV in 2006, states and local providers needed time 
and assistance to modify their data systems and develop measurement approaches to meet 
the new requirements. Recognizing this, the Department took a staged approach to imple- 
mentation and provided multiple technical assistance opportunities to states as they made 


changes to their accountability systems. 


Nonregulatory guidance 
In March 2007, the Department released nonregulatory guidance that outlines preferred 
population and measurement definitions for the secondary and postsecondary core indica- 
tors (Justesen 2007a). This guidance is designed to promote valid, reliable, and comparable 


reporting across states (U.S. Department of Education 2010c). 


Respondents in several case study states expressed appreciation for the guidance and its def- 
initions of populations and performance measures. According to one state administrator, “I 
think that the nonregulatory guidance at this point has been helpful. I think the spirit con- 
tinues to grow among the states to work closely to adopt or interpret it as best we can.” An- 
other said, “I’m a firm believer that the more we can get at common definitions [or] 


common procedures across the country, the better off career tech’s going to be in general.” 


At the same time, respondents valued the flexibility in Perkins IV that allows them to devel- 

op their own definitions for populations and indicators based on data soutces, reporting ca- 

pacity, and program delivery structures. A state administrator noted that, “We met with our 

LEA partners and one of the guiding principles during those discussions was we didn’t want 
a measure that was only important for the Perkins law. ... We wanted to have relevance to 


our institutions ... within the parameters of what was required from the law.” 


? The National Center for Education Statistics in the U.S. Department of Education, Institute of Edu- 
cation Sciences, defines the terms validity and reliability in its Statistical Standards Program (U.S. Depatt- 
ment of Education 2002). Validity is defined as “[T]he extent to which a test or set of operations 
measures what it is supposed to measure. Validity refers to the appropriateness of inferences from test 
scores or other forms of assessment.” Reliability is defined as “[T]he degree to which test scores for a 
group of test takers are consistent over repeated applications of a measurement procedure and hence 
are inferred to be dependable and repeatable for an individual test taker.” 
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Internal capacity, legal barriers, and resource issues, however, pose obstacles to some states’ 
ability to collect and report data, particularly for the placement and technical skill attainment 
indicators (U.S. Department of Education 2010c; U.S. Government Accountability Office 
2009). Further, like Perkins IV, the guidance provides flexibility and includes terminology 
that could be interpreted differently by states. For each indicator, fewer than half of all states 
used the population and measure definitions outlined in the guidance.!° Although the meas- 
ure definitions that states used may be valid and reliable, they may create difficulties in ag- 
gregating information across states. In addition, even when the text of a state’s population or 
measure definition matches the nonregulatory guidance, there may be underlying differences 
in how the state collects and reports data in comparison to other states using the same defi- 
nition. These and other issues are discussed in depth in the Indicators, Populations, and 


Measures section (see p. 98). 


Technical assistance 
To help states transition to and implement Perkins IV, the Department offers technical assis- 
tance opportunities in several forms, including Department-sponsored events and working 
groups, individual state assistance, and a website. The Department invites state CTE direc- 
tors and administrators to participate in national and regional Data Quality Institutes 
(DQIs), which are designed to disseminate information about Perkins reauthorization, assist 
states in developing their performance measurement systems, and promote comparable data 
collection and reporting. DQIs focus on implementing the accountability framework, giving 
states an opportunity to consult with the Department, data and accountability experts, and 


each other about promising practices. 


The Department also convenes periodic conference calls to discuss issues associated with 
the accountability requirements. The Next Steps Work Group (NSWG) is open to staff in all 
eligible agencies and focuses on accountability topics identified by the states and the De- 
partment. Quarterly webinars for state directors provide an opportunity for the Department 
to share information regularly with state administrators. The Perkins Collaborative Resource Net- 
work, a website sponsored by the Department, offers states an online resource for infor- 


mation about Perkins accountability (http://cte.ed.gov). 


The Department has assigned a Regional Accountability Specialist (RAS) to each state. The 
RAS assists states in implementing Perkins program and accountability requirements, prepar- 
ing and submitting state plans and plan updates, and negotiating performance targets. The 
Department also conducts on-site monitoring visits to support states in aligning their poli- 
cies with federal regulations. In addition, since 2005, more than 20 states have applied for 


and received customized technical assistance through a project at the Department.'! This 


10 For more detailed information about states’ use of the nonregulatory guidance, see pages 103 and 104. 
"! Several states have applied for and received customized technical assistance more than once. 
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Department-sponsored initiative provides consultation to individual states to help them de- 
velop their population and measurement definitions and ensure that their state data systems 


meet the new reporting requirements. 


State and local data quality initiatives 
Study findings indicate that administrators and staff in states, IHEs, and LEAs recognize the 
importance of the Perkins IV accountability requirements and are working diligently to en- 


sure that data meet congressional expectations for validity and reliability. 


Data collection 
The six case study states are making progtess in establishing statewide standards and strate- 
gies for collecting high-quality data. While they varied in their approaches and capacity, ad- 
ministrators in each state appeared to understand the accountability requirements and the 


importance of collecting valid and reliable data. 


Two of the six states had data systems that integrated Perkins data with information about 
other education programs. Three more planned to develop comprehensive data systems in 
the future. A state administrator in one of these states observed that “a lot of Perkins report- 
ing and accountability issues will go away and data will get much better” once the state has a 
longitudinal data system. This administrator also noted some of the challenges in implement- 


ing such a system: 


The state’s working right now on a database, if we can ever get all our post- 
secondary [institutions] to cooperate with us. Every student [who] goes to 
public school in [this state] has a state ID number—not the Social [Security 
number], but an assigned unique number. If we can just get . . . all the dif- 
ferent postsecondary institutions to agree [that] everybody uses the same 


number, then we can simply follow those students. 


Local administrators in another case study state relied on software to organize and report da- 
ta. As one administrator explained, the “software generates the numbers and grabs the stu- 
dent data,” but this required substantial set-up, which included “Making the information 
from our student information system available to the software, marking the courses that are 
... CTE courses, and marking the graduates—the twelfth-graders who actually graduated 
because we’re doing this a year in arrears.”” Another local administrator in the same state em- 
ployed an administrative assistant to assemble performance data and enter it by hand into 


the state online data collection system. 


While all six states used some level of technology to make data collection easier and more ef- 
ficient, as well as to improve quality, only one case study state possessed a student-level data 


system capable of showing student outcomes over time and across educational sectors and 
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into the workforce. According to the state director, “We’re able, with all of our longitudinal 
data, to hone in on how our students are doing, how our programs are doing, how individual 


rograms are doing, so that they’re very accountable.” 
g 25 y J 


Some local administrators highlighted steps being taken by their states to collect high-quality 
data. According to a director of an area CTE center, “Definitions and measures are getting 
better at the state level. We have common definitions now, which are the best part. In the 
past, it seemed that rules would change every year because somebody would redefine what 


the definitions were.” 


There were, however, some problems noted. In one state, local administrators felt the state 
did not communicate clearly about accountability requirements. In another state, a local staff 
person said, “It would have helped if there had been some consistency in what our defini- 
tions were. .. . There wasn’t as much collaboration between systems as we should have had 
earlier on and now the definitions have pretty much been mandated to us.” In the same 
state, an LEA representative claimed that the state decided to base its definitions on data 
that ate readily available—and not on “what the feds are really looking for or what would 


have to change in order to get accurate data.” 


Quality control measures 
All states implemented at least one quality control measure to ensure that locally reported 
data are valid and reliable, and many adopted multiple strategies (49 secondary, 44 postsec- 
ondary) (Exhibit 3.2). The most common approaches were technical assistance or guidelines 
on collecting and editing data (45 secondary, 40 postsecondary) and comparing current year 


results to prior year reports (42 secondary, 40 postsecondary). 
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Exhibit 3.2. 
Number of states according to their adoption of quality control measures to ensure validity 


and reliability of local data, by education level 


Secondary Postsecondary 


More than 1 mesure TS . [i . 
Provide technical assistance/ guidelines on 
; eS es 
data collection and editing 
Compare totals to prior year reports IS «2 ee 
Ueda dein | i 2. 
Conicickadotbe! i > | ee ;; 
Other ] 1 | 


No quality control measures 
have been adopted 


No response | 1 | 1 


0 10 20 30 40 50 0 10 20 30 40 50 


Numbet of states Number of states 


Exhibit reads: Secondaty and postsecondary state ditectors adopted multiple quality control measures, including technical assistance or 
guidelines on collecting and editing data (45 secondary, 40 postsecondary) and comparing current year results to prior year reports. 
NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


LEAs and IHEs telied upon state measure guidance to promote validity and reliability (73 
percent of LEAs and 78 percent of IHEs) (Exhibit 3.3).!2 Spot-checks by instructors and 
administrators were another common option for subgrantees at both LEAs and IHEs. Nine 
percent of LEA directors and 8 percent of IHE directors did not use or did not know if they 


had used quality control measures. 


12 All comparisons of the IHE and LEA survey data were tested for statistical significance using the 
Student’s éstatistic, and all differences cited are statistically significant at the p < .05 level. 
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Exhibit 3.3. 


91 


Percentage of LEAs and IHEs according to steps taken to ensure validity and reliability of 


Follow state measure guidance 


Internal review by 
administrators /program chairs 


Review/spot check 
by teachers /administrators 


Provide TA to teachers 


Use software program 
to flag errors 


Other 


No steps taken to ensure 
data accuracy 


Don’t know 


data 


0 20 40 60 80 


Percent 


100 


IHE 


n/a 


n/a 


0 20 40 60 80 


Percent 


100 


Exhibit reads: LEAs and IHEs overwhelmingly relied upon state measure guidance to promote validity and reliability (73 percent LEA, 78 


percent IHE). 
n/a = Not asked. 


! Interpret data with caution. These items have a response rate below 85 percent. 
NOTE: Estimates were weighted to reflect population values. 
SOURCE: LEA and IHE Surveys, 2009. 


In one case study state, a large district reported hiring a part-time team to review the data for 


quality. The local director explained, “We bring in a team of business teachers and other 


teachers who have done this before; they go school-by-school and student-by-student to de- 


termine if these data are correct.” 


Technical assistance 


To support the efforts of LEAs and IHEs to improve both data quality and performance, 


states offered multiple technical assistance opportunities in 2008-09. Nearly all states pro- 


vided more than one technical assistance opportunity (48 secondary, 43 postsecondary) 


(Exhibit 3.4). Only one state reported that it did not provide any technical assistance to its 


secondary subegrantees. 
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Exhibit 3.4. 
Number of states according to types of technical assistance provided to local subgrantees in 
2008-09, by education level 


Secondary Postsecondary 
Provided more than one type of 
none ype of ——— :: «: 
technical assistance 
ee a 2 


technical assistance 


‘sed inc 9 i Ts 


data collection 


Suicide wsining on dota I 11 es» 


collection and reporting 


Hep inc 37 ES 20 


(telephone or e-mail) 


Local site visits i 32 a 3° 
Other Ml 4 mm: 


lon 
aN 
aN 


Did not provide J} 1 0 
technical assistance 
No response | 1 0 
0 10 20 30 40 50 0 10 20 30 40 50 
Number of states Number of states 


Exhibit reads: Nearly all states provided more than one technical assistance opportunity (48 secondary, 43 postsecondary). Only one state 
reported that it did not provide any technical assistance to its secondary subgrantees. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Administrators in each of the six case study states reported providing technical assistance, 
most frequently citing workshops or written guidance. Most also offered assistance via e- 
mail or phone, and several local administrators noted they had used this form of technical 


assistance. 


In one state, state staff reported that they have five full-time help desk attendants who re- 
spond to “questions associated with data uploads, data connections, and data validation,” 
and four regional trainers who “work with school districts all year long.” These staff provide 
technical assistance for several federal programs in addition to Perkins IV. In addition, the 
state hosts an annual conference focused on data quality, where approximately 700 local 
providers attend general and program-specific sessions. A local administrator in this state, 
however, expressed concern about the state’s technical assistance initiatives: “What’s difficult 


for me is that our state doesn’t have a really good data warehouse and a good, current func- 
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tional database for me to pull from. ... It’s the most dysfunctional system Pve ever wit- 
nessed, and I’ve worked in six states. ... The help desk is now dial-a-prayer; there’s nobody 


there to ask for help.” 
In another state, administrators shared their perspectives on data quality initiatives: 


Technical assistance workshops were probably the big ah-ha. We’ve been 
mentioning these measures since we first learned about Perkins IV. ... It re- 
ally didn’t become teal until we had this information out and really got to 
do some of the more personal discussions. We’ve [done] presentations. ... 
We’ve had [the information] published on our website for them. ... We’ve 


distributed it along with the [negotiated state performance levels]. 


Another administrator described how the state assisted local subgrantees who miscoded or 
miscounted students. “We would go one by one, record by record, with the school dis- 
tricts...to see where they were. That’s technical assistance down to the fine grains of sand, 


and it’s an excellent opportunity for them to see the importance of accurate coding.” 


In a state where workshops are the key tool for technical assistance, a state administrator ex- 
plained, “We do an August conference that is aligned around all of the Perkins accountabili- 
ties, assessment development using data to inform instruction, and integrated academics. 
Then we sponsor regional or other statewide pieces throughout the year to reinforce some 
of those things.” An IHE administrator in this state, however, said that “the main thing 
they’ve done is sat in several meetings and worked with us around the definitions and per- 
formance measures to make sure they’re sensible, that we understand them, and that we can 


work with them. Other than that, there’s been very little advance on the technical front.” 


Confidence in data accuracy and completeness 
Most of the secondary state directors were at least somewhat confident that the data report- 
ed for the indicators accurately reflect local performance (Exhibit 3.5). LEA respondents had 
the least confidence in the accuracy of General Educational Development (GED) and state- 
recognized equivalent data (Exhibit 3.6) and some did not know about the accuracy of each 
of these two types of data, which could be due in part to some states not offering these sec- 
ondary completion options. Postsecondary state directors and IHE directors appeared to 
have at least some confidence in the data for most of the indicators, and 47 percent of post- 
secondary local directors were very confident in the accuracy of the data reported for com- 
pletion of industry-recognized credentials, certificates, and degrees (Exhibit 3.7 and Exhibit 
3.8). 
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Exhibit 3.5. 
Number and percentage of state directors according to how confident they were that 


secondary LEA data accurately reflect local performance 


Core indicators 


Academic attainment 


Technical skill attainment 


Completion 
Diploma 
GED 
State-recognized equivalent 
Proficiency credential/ 


cettificate/degree 
Graduation tate 
Placement 
Postsecondary 
education/training 
Military 
Employment 


Nontraditional participation 


Nontraditional completion 


(5-point scale from Not confident to Very Confident) 


Number 
Less than 
Very vety Not 
confident confident confident Don’t 
(5) (2 - 4) () know 
26 23 0 1 
17 29 2 2 
32 17 1 
14 9 3 
6 6 2 
10 14 2 1 
25 20 2 1 
8 37 3 1 
9 30 6 1 
10 32 5 1 
20 28 1 0 


Not 
applicable/ 
No 
response 


35 


24 


Total 
51 


51 


At least 
some 


confidence confidence 


(2-5) 
100 


Percent 


No 


(1) 


12 
14 


13 
11 


94 


Total 
100 


100 


100 


100 


100 


100 


100 


100 
100 
100 


100 


Exhibit reads: Most secondary state directors were at least somewhat confident that the data reported for the indicators accurately reflect 


local performance. 


NOTE: For the secondary completion measure, states responded “Not applicable” for GED (22 states), state-recognized equivalent (34), 
and proficiency credential/certificate/degree (23). Some states did not offer these types of awards or may not have had access to GED 
data. For more information, see the discussion of this measure on p. 122. The percentage calculations exclude “Don’t know” and “Not 


applicable/No response.” 


SOURCE: Secondary State Director Survey, 2009. 
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Exhibit 3.6. 
Percentage of LEA directors according to how confident they were that secondary data 
accurately measure local performance 


(5-point scale from Not confident to Very confident) 


@ Very confident (5) @ Less than very confident (2 - 4) @ Not confident (1) Don’t know 


Academic attainment 


i) 
ioe} 


i) 
nN 
n 


Diploma 


1? TT 3 
Completion State-recognized equivalent 47 


Proficiency credential/certificate /degree 


Graduation rate 


Postsecondaty education/ training 


Placement Military 


Employment 


us 


Nontraditional participation 


RB mp 
A 


Nontraditional completion 


oO 
NO 
oO 
as 
oO 
a 
oO 
[eo] 
oO 
= 
S 


0 


Percent 


Exhibit reads: About half of LEAs were very confident in the accuracy of academic attainment and graduation rate data. 

NOTE: Estimates were weighted to reflect population values. For the secondary completion measure, 30 percent of LEAs responded 
“Don’t know” for GED; 47 percent responded “Don’t know” for state-tecognized equivalent; and 32 percent responded “Don’t know” for 
proficiency credential/certificate/degtee. Not all providers offered these types of awards, and some may not have had access to GED data. 
For more information, see the discussion of this measure on p. 122. 

SOURCE: LEA Survey, 2009. 
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Exhibit 3.7. 
Number and percentage of state directors according to how confident they were that 
postsecondary IHE data accurately reflect local performance 


(5—point scale from Not confident to Very Confident) 


Number Percent 
Less than Not At least 
Very very Not applicable/ some No 
confident confident confident Don’t No confidence confidence 
Core indicators (5) (2-4) (dy know response Total] (2-5) ()) Total 
Technical skill attainment 15 28 3 0 2 48 93 7 100 
Completion 17 28 1 0 2 48 98 2 100 
Retention or transfer 
Retention 21 25 0 0 2 48 100 0 100 
Transfer to BA program 15 26 3 0 4 48 93 7 100 
Placement 
Military 10 24 6 1 7 48 85 15 100 
Apprenticeship 9 20 8 1 10 48 78 22 100 
Employment 14 29 3 0 2 48 93 7 100 
Nonttaditional participation 20 25 2 0 1 48 96 4 100 
Nontraditional completion 


Exhibit reads: Most postsecondary state directors were at least somewhat confident that the data reported for the indicators accurately 
reflect local performance. 

NOTE: For the postsecondary placement measure, some states responded “Not applicable” for military (7) and apprenticeship (10). Some 
states may not have had access to information about these placements. For more information, see the discussion of this measure on p. 129. 
Percentage calculations exclude “Don’t know” and “Not applicable/No response.” 

SOURCE: Postsecondary State Director Survey, 2009. 


CHAPTER 3. ACCOUNTABILITY 97 


Exhibit 3.8. 
Percentage of IHE directors according to how confident they were that postsecondary data 
accurately measure local performance 


(5-point scale from Not confident to Very confident) 


@ Very confident (5) B Less than very confident (2 - 4) @ Not confident (1) Don’t know 


mI 


Technical skill attainment 


Completion 


Retention or transfer 


Miiery eo 
Placement Apprenticeship S27 is | 
Employment 2 es) 7 


Nontraditional participation 


Nontraditional completion 39 48 7 


oO 
NO 
oO 
aN 
oO 
lon 
oO 
co 
oO 
= 
S 


0 


Percent 


Exhibit reads: Forty-seven percent of IHE directors were very confident that data reported for the completion of industry-recognized 
credentials, certificates, and degrees accurately represented local performance. 

NOTE: Estimates were weighted to reflect population values. For the postsecondary placement measure, 18 percent of IHEs responded 
“Don’t know” for both military and apprenticeship. Some institutions may not have had access to information about these types of 
placement. For more information, see the discussion of this measure on p. 129. 

SOURCE: IHE Survey, 2009. 


According to interviews with state and local administrators in the case study states, confi- 
dence in the data varied. In one state, where CTE data are included in a consolidated elec- 
tronic data submission, the state extracted a report and sent it to each district for validation. 
A state administrator reported finding mistakes in local provider data: “Now that we’re re- 
leasing local performance indicators to school districts, we’ve found that a lot of our school 
districts misidentified their students, miscoded in the area of graduation rate. Sometimes ... 
one student can really make or break a school district, so we’ve been working with our dis- 


tricts to help them understand how to identify and code their students.” 
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In another state, where providers used some 20 different data systems, an administrator 
raised concerns about consistency: “There’s no electronic upload, because every high school 
has a different data system. We collect all of our secondary core indicators through a sepa- 
rate online data collection system, and all the LEAs have to hand-enter all that data on en- 
rollments.” In response to a Department inquiry about the validity and reliability of the 
state’s data, the administrator reported, “Our answer was [they’re] about as valid and reliable 


as we can get, because [they’re] the only data we have.” 


A community college administrator reported issues with data consistency between the state 
and local levels. As the administrator explained, “We just last year finally got to the point 
where we saw what the state was ageregating. There’s a mismatch between what we know on 
out campuses for completion rates and what the state is saying our completion rates are. 


We’re trying to figure out how they are pulling those data. The data are messy at this point.” 


A local director in one state had no confidence in the validity and reliability of technical skill 
attainment data. He stated that he “just provided a number above the state target,” and he 
guessed that most of his peers in other local agencies had done the same. He also expressed 
concern about data analysis at the state level, although he had confidence in the data submit- 
ted by local subgrantees: “We submit electronically. [The data are] manipulated behind 
closed doors, and [they come] back out different from what was submitted. ... You could 
just look at the data and see that they weren’t right, didn’t add up. They were clearly bogus, 
didn’t pass the smell test.” He acknowledged recent improvements, however, stating that the 


state director “has done an awful lot of work on getting higher quality data.” 


Indicators, Populations, and Measures 


Perkins IV outlines six secondary and five postsecondary core indicators. The Department’s 
nonregulatory guidance outlines two student populations for each sector as well as eight 
secondary and six postsecondary performance measures aligned with the Perkins core indi- 
cators (Exhibit 3.9). The guidance offers more specificity for each indicator by splitting some 
indicators into multiple measures (such as academic attainment) and provides direction for 
states in determining their measurement approaches. Discussion of the individual measures 
in this section explores how states are implementing the core indicators, populations, and 


performance measures. 
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Exhibit 3.9. 
Perkins IV core indicators and performance measures from the Department’s 
nonregulatory guidance 


Core indicators Performance measures 


Secondary 


; , 1S1: Academic attainment—Reading/Language Arts 
Academic attainment 


: Academic attainment—Mathematics 


: Nontraditional participation 
Nontraditional participation and completion 
6S2: Nontraditional completion 


Postsecondary 

Technical skill attainment 1P1: Technical skill attainment 

Completion =—s—(<‘ se!!!” Credential, certificate, or degree 
Retention or transfer = ttsti(‘é#!©©©©© SPL Retentionortransfer =—00000t—i(‘séSCS~™S 
Placement =  4PL:Placement 


. ae ; 5P1: Nontraditional participation 
Nontraditional participation and completion 
5P2: Nontraditional completion 


Exhibit reads: Perkins IV outlines six secondary and five postsecondary core indicators. The Department’s nonregulatory guidance 
describes eight secondary and six postsecondary performance measures that align with the Perkins indicators. 
SOURCE: The Carl D. Perkins Career and Technical Education Act of 2006; Justesen (2007a). 


The survey results suggest that most states employ population and measure definitions that 
blend nonregulatory guidance with state-developed approaches (Exhibit 3.10 and Exhibit 
3.11). Although the methodologies that states use may result in valid and reliable data within 
states, variability in how states define their populations and measures reduces the possibility 


of validly aggregating data at the national level. 


13 For the purposes of this report, postsecondary performance measure 2P1, “Credential, certificate, 
or degree” is referred to as “Completion.” 
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In 20 states, secondary state directors reported that they consulted the guidance when defin- 
ing each secondary population and measure,'* while those in nine states reported using the 
guidance verbatim for every secondary population and measure definition (Exhibit 3.10). 


Every state reported that they consulted the guidance or used it verbatim for at least one 


secondary population or measure. 


Exhibit 3.10. 


100 


Number of states according to the extent to which they used the nonregulatory guidance to 


craft secondary population and performance measure definitions 


Populations and performance measures 
Populations 

CTE participants 

CTE concentrators 


Use of guidance for both populations 


Measures 
Academic attainment (1S1, 182) 
Technical skill attainment (2S1) 
Secondary school completion (381) 
Graduation rate (481) 
Placement (581) 
Nontraditional participation (6S1) 
Nontraditional completion (682) 


Use of guidance for all measures 


Use of guidance for every population and measure 


Used 


verbatim 


18 
14 
14 


Consulted Did not use Don’t know 


30 
35 
30 


27 
32 
30 
27 
30 
27 
29 
21 


20 


Secondary 


SN KN RP RFP RP FP OO 


0 


eo oc. ocUcUcCOlhUhrRhhUO 


0 


Total 


51 
51 


O Of OF OW Oo MH WM 
- 


Exhibit reads: Twenty states consulted the guidance when defining secondary populations and measures, while nine used the guidance 


verbatim for every secondary population and measure definition. Every state consulted or used the guidance verbatim for at least one 


secondary population or measure definition. 
SOURCE: Secondary State Director Survey, 2009. 


The survey did not specify the level of consultation, which potentially could vary from changing a 
few words to considering the guidance without using it. 
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Among the 48 states represented in the postsecondary survey results, 14 consulted the guid- 


ance for every population and measure, although only five states used the guidance verbatim 


for every postsecondary population and measure (Exhibit 3.11). All states consulted or used 


the guidance verbatim for at least one postsecondary population or measure definition. 


craft postsecondary populations and performance measure definitions 


Populations and performance measutes 


Populations 
CTE participants 
CTE concentrators 


Use of guidance for both populations 


Measutes 
Technical skill attainment (1P1) 
Completion (2P1) 
Retention or transfer (3P1) 
Placement (4P1) 
Nontraditional participation (5P1) 
Nontraditional completion (5P2) 


Use of guidance for all measures 


Use of guidance for every population 


and measure 


Exhibit 3.11. 
Number of states according to the extent to which they used the nonregulatory guidance to 


Used verbatim Consulted 


5 


25 
33 
25 


29 
28 
26 
26 
19 
21 
16 


14 


Postsecondary 


Did not use 


oR FP OF FP wD 


0 


Don’t know 


[>  ) 


0 


Total 


48 
48 


48 
48 
48 
48 
48 
48 


Exhibit reads: Fourteen states consulted the guidance for every postsecondary population and measure, although only five used it 
verbatim for every population and measure definition. All states consulted or used the guidance verbatim for at least one postsecondary 


population or measure definition. 


SOURCE: Postsecondary State Director Survey, 2009. 
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Case study states: 
Perspectives on nonregulatory guidance 


During the case study visits, interviewers spoke with state administrators regarding their 
views of the nonregulatory guidance and whether they thought the Department should re- 
lease regulations for the Perkins accountability requirements. 


On the current nonregulatory guidance: “| didn’t use the nonregulatory guidance. | went 
to [our eligible agency] for my guidance at every level, every step. The negotiations 
for all of our core indicators were done collaboratively between us and our [eligible 
agency]. | took a look at their nonregulatory guidance, but that’s pretty much what | 
did with it, because there really wasn’t much there, to be honest with you.” 


“| don’t know how helpful it was. To me, it seemed like the nonregulatory guidance 
was really structured off the Data Quality Institutes. The problem with that was that 
there were many small states there who had an equal voice. . . .| guess | would rather 
have the nonregulatory guidance than not have it, but there are issues with it. | have 
asked OVAE ‘What's the question you want answered?’ and they can’t tell me. The 
only problem with the core indicators is that | don’t think they measure the right things. 
| think some of the core indicators are just giving Congress data, but they’re not nec- 
essarily giving them worthwhile data, anything they can really use.” 


On releasing regulations: “Yes and no. Yes, in the spirit of trying to create a common 
comparative approach, but no, because it typically would mean that it doesn’t come 
with funds to help implement systems that would get us there.” 


“OVAE should have at least nonregulatory guidance in order for there to be some 
consistency across the states. Without such guidance, it leaves each state on its own 
to interpret the legislation and to decide on how it wants to build its measures. How- 
ever, the guidance should not be developed in a vacuum. OVAE should convene a 
committee of state directors to work with them in developing the guidance.” 


“| think detailed guidance is better, since federal regulations are often not appropriate 
for shared-time systems in career and technical education.” 


“The reason | don’t want regulatory guidance is the nonregulatory guidance | don't 
agree with. So | don’t want it to be regulation. If |’ve got the same kind of regulations 
that | have as nonregulatory guidance. . . heaven help us.” 
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Student Populations 


Por Perkins IV, the Department defined two student populations—CTE participants and 
CTE concentrators—for both secondary and postsecondary education. The legislation also 
requires states to report disaggregated results for seven secondary and six postsecondary 


special populations. 


Participants and concentrators 
Generally, participants are students who receive at least a minimal level of CTE instruction, 


and concentrators ate students who participate more extensively in CTE coursework 


(Exhibit 3.12). 


Exhibit 3.12. 
Nonregulatory guidance: Secondary and postsecondary populations 


Secondary populations 


CTE participant A secondaty student who has earned one (1) or more credits in any career and technical 
education (CTE) program area. 


CTE concentrator A secondary student who has earned three (3) or more credits in a single CTE program 
area (e.g., health care or business services), or two (2) credits in a single CTE program 
area, but only in those program areas where 2 credit sequences at the secondary level are 
recognized by the State and/or its local eligible recipients. 


Postsecondary/Adult populations? 


CTE participant A postsecondary/adult student who has earned one (1) ot more credits in any CTE pro- 
gram area. 
CTE concentrator A postsecondary/adult student who: (1) completes at least 12 academic or CTE credits 


within a single program area sequence that is comprised of 12 or more academic and 
technical credits and terminates in the award of an industry- recognized credential, a cer- 
tificate, or a degree; or (2) completes a short-term CTE program sequence of less than 12 
credit units that terminates in an industry- recognized credential, a certificate, or a degree. 


Exhibit reads: According to the Department’s nonregulatory guidance, participants are students who receive at least a minimal level 
of CTE instruction, and concentrators are students who participate more extensively in CTE. 

4 Perkins IV does not refer to an “adult” population for accountability reporting; however, the Department’s nonregulatory guidance 
uses the language “postsecondary/adult” in the population definitions. Several states offer adult CTE programs outside their 
community and technical colleges. For 2007-08, six states—Florida, Louisiana, Ohio, Oklahoma, Rhode Island, and Tennessee— 
negotiated separate performance targets and reported separate data for their adult programs (U.S. Department of Education 2010c). 
The remainder of this chapter will refer only to “postsecondary.” 

SOURCE: Justesen (2007a). 


The Department requires states to report on the number of participants served in aggregate, 
as well as disageregated by special population. Concentrators are the cohort of students 
evaluated in every secondary and postsecondary performance measure. The only exception is 
the nontraditional participation measure (6S1 for secondary, 5P1 for postsecondary), which 


assesses the outcomes of participants instead of concentrators. 


CHAPTER 3. ACCOUNTABILITY 104 


The Department developed and disseminated the nonregulatory guidance to promote validi- 
ty, reliability, and comparability in state and local performance data (U.S. Department of Ed- 
ucation 2010c). To achieve comparability, evaluations of student outcomes should apply 
similar performance measure definitions to similar cohorts of students—in this case, concen- 
trators. Only 14 secondary and 13 postsecondary state directors, however, reported using the 
concentrator definition verbatim from the nonregulatory guidance (Exhibit 3.10 and Exhibit 
3.11), suggesting that definitions of concentrators, and therefore concentrator cohorts, may 


vaty across states. 


In its 2007-08 Report to Congress on State Performance, the Department noted similar concerns. 
The specific issue was reporting on students at different points along their educational path: 
some states include students relatively new to CTE, while other states report on students 
who are close to completing, or have completed, their CTE programs. As the Department 
observed, there can be no expectation that students at such different stages in their programs 
would have similar levels of attainment, making comparisons of these different groups inva- 
lid (U.S. Department of Education 2010c). 


The case studies revealed that at least three states continue to designate a student “complet- 
er” population at one or both secondary and postsecondary levels.'> In Perkins III, some 
states used the completer population to identify students who actually complete a CTE pro- 
gram or graduate. Because the nonregulatory guidance does not mention this population, 
these states’ use of completers—instead of concentrators—as the student cohort evaluated 


in some indicators will reduce comparability. 


Besides these differences in concentrator definitions, states and local subgrantees also vary in 
how they determine which students are concentrators. The decision to identify a student as a 
patticipant or concentrator is made at the local level in some states, while others make that 

determination for all students at the state level. According to survey responses, 26 percent of 
LEA directors and 34 percent of IHE directors reported that their state identified concentra- 
tors using a state-level database (Exhibit 3.13). The most common method for both LEAs 

(57 percent) and IHEs (60 percent) was having local administrators use local data systems to 


identify concentrators. 


'5 One case study state provided oral definitions of CTE completers, but the definitions are not in- 
cluded in its Perkins five-year plan. 
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Exhibit 3.13. 
Percentage of LEAs and IHEs that used various methods to identify concentrators 


LEA IHE 


Administrators identify 


students using local management rs 57 iss 60 


information system 


Teachers /faculty identify students 
bcd oncoucckiny nn 2 a 
State identifies students using 


statewide database; no action at | | 26 P| 34 


LEA/IHE 
Students self-report 
concentrator status || 12 | 20 
Other 1 31 a 4! 


Don’t know a 9 | 5 


40 60 80 100 0 20 40 60 80 100 


Percent 


Oo 
NO 
Oo 


Percent 


Exhibit reads: In more than half of LEAs and IHEs, local administrators use local data systems to identify concentrators. 
! Interpret data with caution. These items have a response rate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


Given that so few states used the nonregulatory guidance verbatim to define a concentrator, 
coupled with the varied methods used to identify concentrators, it is probable that concen- 
trator cohorts differ across states. Some of the administrators interviewed during the case 
studies explained their reasons for developing different definitions, several of which related 
to variations in how they administer programs. One said, “We couldn’t just take the concen- 
trator at the 12-credit level. We have completers at 10-credit levels, so we had to fit things 
like that into how our system works.” In another state, a secondary administrator offered, 
“We didn’t use [the Department’s] recommended definitions because we don’t use Carnegie 


units; it was easier for us to use hours.” 


The following sections offer a few additional reasons for states’ use of a definition other 
than that provided in the nonregulatory guidance, including timeframe, data system capacity, 


and challenges linking students to CTE programs. 
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Clarity of timeframe 
In the nonregulatory guidance, the concentrator definition does not indicate a timeframe 
during which students must earn CTE credit or be enrolled in CTE coursework to qualify as 
a concentrator. It is therefore unclear if states are looking only at students enrolled in the 
current reporting year!® to identify concentrators, or if they are including students who 
reached the concentrator threshold in prior years, regardless of those students’ current en- 


rollment. 


Among the six case study states, for example, two states used the nonregulatory guidance 
verbatim to define a secondary concentrator. The definition designated a secondary concen- 
trator as a student “who has earned” a specific threshold of credits in a single CTE program 
area, but it did not specify when those credits should be earned or that the student be en- 
rolled in a CTE course during the reporting year (Exhibit 3.12). A third state used a slightly 
different definition, but still based it on credits earned, not current enrollment. These states, 
therefore, may have included students in their concentrator cohorts who achieved concen- 
trator status in previous years, but who were not concentrators in the reporting year (e.g., a 
student who achieved concentrator status in grade 11, who was not enrolled in CTE in grade 


12, but was included in the performance outcomes in that reporting year). 


Another state referred to students “who have enrolled” in CTE, although it did not appear 
to require current enrollment. The last two states required students to be currently enrolled 
in addition to having completed a threshold number of hours or proportion of a program 
sequence. These states likely did not include students who achieved concentrator status in a 
previous year, unless they were enrolled in the reporting year. While each of these approach- 
es may offer states valuable information about their students, they result in dissimilar con- 


centrator cohorts across states. 


Data system capacity 
Data system capacity varies among states, affecting the ability of some states to identify stu- 
dent populations and in which, if any, CTE program a student is enrolled. A small number 
of states did not have the ability to monitor students through grade 12 (9 secondary) or over 
time (14 postsecondary) (Exhibit 3.14), suggesting that they were unable to review prior year 
records to see if a student had accumulated the credits or courses needed to reach the con- 


centrator threshold. 


'6 The reporting year is the program year ending on June 30 prior to the December 31 Consolidated 
Annual Report (CAR) submission deadline. For the December 2010 CAR submission, the reporting 
year was 2009-10 (July 1, 2009, through June 30, 2010). 
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Exhibit 3.14. 
Number of states with the ability to monitor individual students 
with Perkins database, by education level 


Don’t No 
Education level Yes No know response Total 
Secondary: Monitor individual students through grade 12 40 9 1 1 51 
Postsecondary: Monitor individual students over time 28 14 5 1 48 


Exhibit reads: A small number of state directors reported that they did not have the ability to monitor students through grade 12 (9 
secondary) or over time (14 postsecondary). 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Another complicating factor arises from postsecondary students enrolling in multiple institu- 
tions to get the courses they need. According to a 2006 report, among undergraduate stu- 
dents who entered postsecondary education shortly after leaving high school, nearly 60 
percent attended more than one institution (Adelman 2006). Without a state system with the 
capacity to track individual students across institutions, match coursetaking records, and 
identify programs at the state level, a student taking elements of a CTE program at different 
institutions may be considered a participant at the individual institutions, but might in fact be 


a concentrator when viewed across institutions. 


Linking students to CTE programs 
Even states with data systems that can track students over time may encounter challenges in 
trying to determine if students are enrolled in a CTE program, and if so, which program. 
Programs sometimes include similar coursework or build upon one another. An example is 
health-related coursework, which could apply to multiple programs such as Dental Assistant, 
Certified Nursing Assistant, Licensed Practical Nurse, and Registered Nurse, among others. 
Programs also may offer students some flexibility in the order of courses they take. States 
noted the nonlinear nature of postsecondary coursework during the 2005 Data Quality Insti- 
tute, suggesting that some states may find it difficult to determine the program in which a 
student is enrolled (Data Quality Institute 2006a). 


At the postsecondary level, there are additional challenges to determining what program or 
goal a student is pursuing. Students enroll in community and technical colleges for a variety 
of reasons, including seeking a degree or certificate, improving basic literacy and numeracy 
skills, developing career-related skills, pursuing personal enrichment (Bailey, Leinbach, and 
Jenkins 2006), and “sampling” college (Adelman 2005). Colleges vary in their approaches to 
capturing student “intent” (.e., students’ reasons for attending and the educational path they 
intend to pursue). Some institutions ask students to self-report their intent upon enrollment 
(Offenstein and Shulock 2009), but may not require students to declare a specific program or 


major. Students who change their plans after initially reporting their intent may not update 
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their information over time, and the institution may be unaware of the change until the stu- 
dent applies for a degree or certificate. A 2006 report from the Community College Research 
Center (Bailey, Leinbach, and Jenkins 2006) suggested that, while there was a relationship be- 


tween students’ primary reasons for enrolling and student outcomes: 


A student’s stated primary reason for enrolling did not preclude achieve- 
ment of other objectives. Many students who stated that their primary rea- 
son was “job skills” or “personal enrichment” still earned a certificate or 
degree or transferred, suggesting that many of them sought to attain the 
non-degree goals by earning postsecondary credentials, not necessarily just 


by taking a limited number of courses (p. 9). 


For states that rely on student intent to determine what program a student is in, the self- 
reported intent information captured when a student enrolls may be transferred to the state 
data system, but it still could be inaccurate for some students. States raised this issue during 
the June 2005 DQI, noting that it is difficult to evaluate the accuracy of student intent across 
institutions and states (Data Quality Institute 2006a). 


Special populations 
Perkins IV requires states to report performance data about several special populations, in- 
cluding individuals with disabilities; individuals from economically disadvantaged families, 
including foster children; individuals preparing for nontraditional fields; single parents, in- 
cluding single pregnant women; displaced homemakers; and individuals with limited English 
proficiency (Sec. 3(29)). Perkins IV also requires each eligible secondary recipient to report 
indicator data annually, disaggregated by the categories of students listed in ESEA Sec. 
1111(h)(1)(C)\@. While most of the ESEA categories are identical to the Perkins IV special 
populations, E.S'EA also includes “migrant status.” 


Most secondary and postsecondary state directors had at least some confidence in the accu- 
racy and completeness of the special population data reported by local subgrantees (Exhibit 
3.15). Secondary state directors were most confident in the data for students with disabilities 
and least confident in the data reported for single parents and single pregnant women. For 
postsecondary education, state directors had the greatest confidence in the accuracy and 
completeness of data reported for economically disadvantaged and foster children and stu- 
dents preparing for nontraditional fields. Postsecondary state directors were less confident in 


the data reported for students with disabilities than their secondary peers. 
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Exhibit 3.15. 
Number and percentage of state directors according to how confident they were in the 
accuracy and completeness of special populations data reported by local subgrantees, by 


education level 


(5—point scale from Not confident to Very Confident) 


Number Percent 
Less than Not At least 
Very very Not applicable / some No 
confident confident confident Don’t No confidence confidence 
Special populations (5) (2 - 4) () know response Total} (2-5) (1) Total 
Secondary 
Students with disabilities 30 19 1 0 1 51 98 2 100 
Economically disadvantaged, 
foster children 24 24 2 0 1 51 96 4 100 
Students preparing for 
nontraditional fields 21 25 2 2 1 51 96 4 100 
Single parents, 
single pregnant women 10 27 9 2 3 51 80 20 100 
Displaced homemakers 7 19 5 1 19 51 84 16 100 
Students with limited 
English proficiency 23 25 2 0 1 51 96 4 100 
Migrant students 12 25 6 1 7 51 86 14 100 
Postsecondary 
Students with disabilities 14 25 7 0 2 48 85 15 100 
Economically disadvantaged, 
foster children 12 28 6 0 2 48 87 13 100 
Students preparing for 
nontraditional fields 18 22 6 0 2 48 87 13 100 
Single parents, 
single pregnant women 5 32 8 1 2 48 82 18 100 
Displaced homemakers 5 28 12 1 2 48 73 27 100 
Students with limited 
English proficiency 14 23 vi 0 4 48 84 16 100 


Exhibit reads: Secondary state directors were most confident in the special populations data for students with disabilities (49). 
Postsecondaty state directors had the greatest confidence in the accuracy and completeness of data for economically disadvantaged/foster 
children (40) and students preparing for nontraditional fields (40). 

NOTE: Percentage calculations exclude “Don’t know,” and “Not applicable/No response.” 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Most LEA and IHE directors had at least some confidence in the accuracy of data they re- 


ported for any special population (Exhibit 3.16). About 30 percent of LEA respondents did 


not know if the data reported for displaced homemakers were accurate, although this may be 


due to the smaller numbers of displaced homemakers served by secondary institutions. 


About one-quarter indicated they did not know if the data for migrant students were accu- 


rate. 


Special populations 
Students with disabilities 
Economically disadvantaged, 
foster children 
Students preparing for 
nontraditional fields 
Single parents, 
single pregnant women 
Displaced homemakers 
Students with limited 
English proficiency 


Migrant students 


Exhibit 3.16. 
Percentage of LEA and IHE directors according to how confident they were in the accuracy 


of the data reported for special populations 


(5—point scale from Not confident to Very Confident) 


LEA 
Less than 
Very vety Not 
confident confident confident Don’t 

(5) (2-4) (1) 
53 38 2 
48 42 2 
35 51 4 
33 43 7 
24 37 10 
46 38 3 
36 36 4 


know Total 


100 


IHE 
Less than 
Very very Not 
confident confident confident Don’t 
(5) (2 - 4) (4) know Total 
33 54 6 6 00 
32 55 7 6 100 
34 55 6 6 00 
23 56 12 9 00 
22 54 14 10 00 
27 58 8 8 100 
n/a n/a n/a n/a n/a 


Exhibit reads: Among secondary local directors, 53 percent were very confident in the accuracy of data reported for students with 


disabilities, while only 33 percent of postsecondary local directors reported the same. Less than 35 percent of IHE directors were very 


confident about the accuracy of any special population data. 


n/a Not available. 


NOTE: Estimates were weighted to reflect population values. 


SOURCE: LEA and IHE Surveys, 2009. 


Case study respondents at the state and local levels reported efforts to collect special popula- 


tions data, while indicating some concern about obtaining valid and reliable data for all spe- 


cial population categories. A postsecondary administrator said that “special populations data 


are usually collected through self-reporting. For the most part, I think they’re valid, but I 


think no one understands what a displaced homemaker is.” Another respondent noted that 


an individual’s status as part of a special population may change: “Because you can be [in 


one of] those [categories] one semester and not another semester, [the IHEs] have to ask 


those [questions] every semester.” 
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A secondary district administrator related problems obtaining reliable information about 
some special populations: “Foster care ... those are hard data to get... unless you just ask 
the student. [They’re] not reported... . Pregnant teen? ... And how do you ask that in a 
class?” In a local postsecondary institution that collects some special populations data 
through student self-report, an administrator said, “any student who self-identifies in a cate- 
gory is easy to track,” acknowledging at the same time that students with disabilities “may 
not self-identify because they don’t want to share; they may not self-identify because they 


don’t know.” 
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Reporting special populations data for postsecondary 
students: NRCCTE report 


Findings from a study conducted for the National Research Center for Career and Tech- 
nical Education (NRCCTE) shed additional light on how some states report special popu- 
lations data for postsecondary students (Kotamraju, Richards, Wun, and Klein 2010). The 
12 states participating in the study noted several issues affecting their ability to collect and 
report data on special populations, including uncertainty around collecting certain types of 
student information. NRCCTE study findings suggest that there may be variability in how 
states report postsecondary data on special populations. 


Students with disabilities: Some study states mentioned reluctance on the part of insti- 
tutions to share information about student disabilities as a challenge to collecting 
these data. Other states questioned whether students should be identified through a 
documented disability or by self-report. 


Economically disadvantaged students: Pell Grant receipt was the most common meth- 
od study states used to identify economically disadvantaged students. States noted 
that this approach did not capture students who chose not to apply for financial aid or 
who did not qualify for federal aid because they were enrolled less than half-time. 
Several, but not all, states sought additional data to identify economically disadvan- 
taged students, including data from the Bureau of Indian Affairs and public assistance 
programs as well as foster child status. 


Students with limited English proficiency (LEP): Some study states categorized stu- 
dents as LEP if they had ever taken an English as a Second Language (ESL) class, 
while others looked only at ESL coursetaking in the reporting year. Some states 
based LEP status on whether students spoke a language other than English at home, 
and still others relied on assessments to identify LEP students. As indicated in the re- 
port: “States agreed that if policymakers provided greater clarity on this definition, 
they would likely be able to adapt their individual data collection and reporting to be 
more comparable across states” (Kotamraju et al. 2010, p.12). 


Displaced homemakers and single parents: The states did not all ask students if they 
were parenting or whether they were displaced homemakers. The study mentioned 
tracking individuals entering from programs targeted to displaced homemakers or 
single parents, but indicated that most study states did not have reliable methods to 
collect this information or did not have such programs. 


Core Indicators and Performance Measures 


States must report their performance and progress toward negotiated targets for each core 
indicator, both in the aggregate and disaggregated by special populations. This section dis- 


cusses each core indicator in depth. 
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Academic attainment 
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Perkins IV includes academic attainment as an indicator only for secondary education, elimi- 


nating it for postsecondary education.'’ The Department’s nonregulatory guidance splits the 


secondary indicator into two separate measures: reading/language arts and mathematics 


(Exhibit 3.17). 


Exhibit 3.17. 


Nonregulatory guidance: Academic attainment 


Secondary 


1S1: Academic attainment— 
Reading/ Language Arts 


Numerator: Number of CTE concentrators who have met the proficient or advanced 
level on the Statewide high school reading/language arts assessment administered by 
the State under Section 1111(b)(3) of the Eementary and Secondary Education Act (ESEA) as 
amended by the No Child Left Behind Act based on the scores that were included in the 
State’s computation of adequate yearly progress (AYP) and who, in the reporting year, left 
secondary education. 

Denominator: Number of CTE concentrators who took the ESEA assessments in read- 
ing/language arts whose scores were included in the State’s computation of AYP and 
who, in the reporting year, left secondary education. 


182: Academic attainment—Mathematics 


Numerator: Number of CTE concentrators who have met the proficient or advanced 
level on the Statewide high school mathematics assessment administered by the State 
under Section 1111(b)(3) of the Elementary and Secondary Education Act (ESEA) as amended 
by the No Child Left Behind Act based on the scores that were included in the State’s com- 
putation of AYP and who, in the reporting year, left secondary education. 

Denominator: Number of CTE concentrators who took the ESEA assessments in 
mathematics whose scores were included in the State’s computation of AYP and who, in 
the reporting year, left secondary education. 


Exhibit reads: The Department’s nonregulatory guidance splits the secondary academic attainment indicator into two separate 
measures: reading/language arts and mathematics. 


SOURCE: Justesen (2007a). 


States appear to have confidence in the accuracy and completeness of the data reported by 


local providers for academic attainment. Among secondary state directors, 26 were very con- 


fident that the data accurately reflect performance, and none reported no confidence in the 


data (Exhibit 3.5). LEA directors appear similarly confident, with only 3 percent reporting 


no confidence at all and 46 percent reporting that they were very confident in the accuracy 


of their academic attainment data (Exhibit 3.6). 


Some states and local subgrantees registered uncertainty regarding this indicator, however, 


with 23 state directors and 44 percent of LEA directors responding that they had some con- 


'7 According to a report from the June 2005 DQI, states were in nearly unanimous agreement that ac- 
ademic skill attainment should be removed from the postsecondary core indicators, contending that 
CTE programs have different purposes and outcomes at the secondary and postsecondary levels, and 
the results of the indicator are duplicated in other measures, such as completion and placement (Data 
Quality Institute 2006a). 
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fidence but were less than very confident about the data reported for the measures. There 
are several reasons that directors might question the accuracy and completeness of their data, 
some of which also may affect comparability of data across states. These include the timing 
of assessments, identifying concentrators who have left education, and applying the ESEA 


methodology to a subset of students, which are discussed below. 


Timing of assessments 
While some students take the tests in 9th or 10th grade, others are assessed in 11th or 12th 
grade. As of 2007-08, according to the Center on Education Policy, 18 states offered their 
ESEA assessment in 10th grade and 14 offered it in 11th grade. The remaining states did 
not specify when assessments were offered (2) or reported assessing students over multiple 
years, ranging from 9th to 12th grade, depending on the subject (16) (Center on Education 
Policy 2010). 


The decision to alter the construct of this indicator to match the ESEA reporting method- 
ology may indicate an interest in understanding the effect of CTE coursework on students, 
particularly whether students in CTE programs fare better or worse than all students on 
statewide assessments. For students who begin CTE coursework in 11th or 12th grade, 
however, the ESEA assessment may occur before they have much, if any, exposure to CTE. 


According to an administrator in one case study state, 


In our state, we give an exit exam in the 10th grade... and the majority of 
students who enroll in CTE don’t even do so until their junior year. In some 
LEAs, that could go against kids. Why would I enroll a low-performing stu- 
dent in CTE when I know two years from now you're going to look at that 


measure and hold it against me? It really is a nonsensical measure. 
In another state, an administrator explained, 


The assessment is administered in October of the 11th grade and measures 
end-of-10th-grade learning, but the students typically begin their CTE core en- 
rollment at the beginning of grade 11. So the regional CTE centers are held 
accountable for those scores when that learning occurred before they even en- 
rolled in the centers. Use of the [ESEA] measurement is not measuring aca- 


demic gain; it only measures differences in successive cohorts of students. 


Identifying concentrators who leave 
The nonregulatory guidance calls for states to assess students at the time they leave second- 
ary education. Students leave high school at different times and for different reasons, how- 
ever. Some students graduate after four years, while others graduate early or late. Still others 


leave without graduating because they are transferring to another school or dropping out al- 
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together. States that follow the guidance must be able to differentiate among these different 
types of student departures to determine which students are “leavers” in order to report aca- 


demic attainment accurately. 


States also must be able to match concentrators who leave in a reporting year with their as- 
sessment results, regardless of whether the students took the assessment that year. In addi- 
tion, some students may leave without taking the statewide assessment. Depending on when 
an assessment is offered, some students may graduate or leave high school in a different year 
than the one in which they took the exam. For example, the 32 states that offered their 
2007-08 ESEA assessments in the 10th or 11th grade must match the assessment results 
from prior years to records for CTE concentrators graduating at the end of 12th grade. 
States that reported being unable to monitor students through the 12th grade may face chal- 


lenges in matching student concentrator records with assessment records from prior years. 


Applying the ESEA methodology 
There is a small body of evidence indicating that some states may be having difficulty ensur- 
ing that the methodology used to calculate academic attainment for ESEA is accurately ap- 
plied for Perkins IV accountability (U.S. Government Accountability Office 2009; Richards 
2009; Richards and Schoelkopf 2008). States with a Perkins data collection system that is sep- 
arate from other state secondary data systems must replicate the ESF.A computation used to 
calculate results for all students when analyzing data only for CTE concentrators. The as- 
sumptions inherent in the “all students” methodology may pose challenges when applied to 
another data system with different assumptions, separate reporting requirements, and a sub- 


set of students. 


Technical skill attainment 
Perkins IV retains the technical skill attainment indicator for both the secondary and post- 
secondary levels and adds a requirement that technical skill proficiencies be aligned with in- 
dustry-recognized standards, if available and appropriate. The legislation describes this core 
indicator as “Student attainment of challenging career and technical skill proficiencies, in- 
cluding student achievement on technical assessments, that are aligned with industry- 


recognized standards, if available and appropriate.” 


In addressing the new language, the Department’s nonregulatory guidance focuses more 
closely on technical assessments than it does on skill proficiencies, asking states to report the 
number of concentrators who pass technical skill assessments (TSAs) aligned with industry- 


recognized standards, if available and appropriate (Exhibit 3.18). 
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Exhibit 3.18. 
Nonregulatory guidance: Technical skill attainment 


Secondary 

2S1: Technical skill attainment Numerator: Number of CTE concentrators who passed technical skill assessments that 
are aligned with industry-recognized standards, if available and appropriate, during the 
reporting year. 
Denominator: Number of CTE concentrators who took the assessments during the 
reporting year. 

Postsecondary 

1P1: Technical skill attainment Numerator: Number of CTE concentrators who passed technical skill assessments that 


are aligned with industry-recognized standards, if available and appropriate, during the 
reporting year. 


Denominator: Number of CTE concentrators who took technical skill assessments dur- 


ing the reporting year. 


Exhibit reads: The Department’s nonregulatory guidance asks states to report the number of concentrators who pass technical skill 
assessments (TSAs) aligned with industry-recognized standards, if available and appropriate. 
SOURCE: Justesen (2007a). 


Seventeen secondary and 14 postsecondary state directors reported using the nonregulatory 
guidance verbatim for the technical skill attainment measure, while 32 secondary and 29 
postsecondary directors reported consulting the guidance (Exhibit 3.10 and Exhibit 3.11), 
suggesting there is variation in how technical skill attainment data are collected and reported 
among states. The six case study states varied in their definitions as well. Only one used the 
nonregulatory guidance; the other five developed different measures for both the secondary 


and postsecondary levels. 


While very few states indicated that they had no confidence in the accuracy of technical skill 
attainment data (Exhibit 3.5 and Exhibit 3.6), according to the U.S. Government Accounta- 
bility Office (GAO), 38 states at the secondary level and 39 states at the postsecondary level 
faced great or very great challenges in collecting data on technical skill attainment (U.S. Gov- 
ernment Accountability Office 2009). Some of those challenges may include allowable and 
available methods for collecting data, timing of measurement, cost of assessments, and ac- 
cess to data (U.S. Department of Education 2010c; Hilber and Sheets 2007; Stanley 2008; 


DeWitt 2008), and are discussed in more detail in the next sections. 


Methods for measuring attainment 
The study’s survey results and a recent GAO report indicate that states are using multiple 
methods for assessing technical skill attainment (U.S. Government Accountability Office 
2009). According to the survey results, most states indicated that at least some of their LEAs 
used industry-developed and employer-validated exams to assess students’ technical skill at- 
tainment (Exhibit 3.19). While 10 secondary state directors reported that all their LEAs used 
program completion, 22 indicated that none of their LEAs used that approach. Of the post- 
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secondary state directors responding to the question, 39 indicated that at least some IHEs 
used state licensing and credentialing exams to assess postsecondary technical skill attain- 


ment. Only seven states reported that any IHEs used state-developed skill exams. 


Exhibit 3.19. 
Number and percentage of states according to the extent to which local subgrantees used 
various tools to assess technical skill attainment in 2008-09, by education level 


Number Percent 
Don’t No At least 
Assessment tools All Many Some None know response Total| some None Total 
Secondary 
Industry developed, employer 
validated exam 4 14 25 4 3 5 91 9 100 
National licensing/credentialing exam 2 11 26 3 5 83 17 100 
Commercially developed exam 3 8 26 10 3 1 51 79 21 100 
State licensing/credentialing exam 2 11 24 10 i) 1 5 79 21 100 
Locally developed skill exam 5 3 17 19 6 5 57 43 100 
CTE course or program completion 10 7 7 22 4 1 51 52 48 100 
State developed skill exam 9 5 5 31 0 1 51 38 62 100 
Grade point average 8 5 4 30 3 5 36 64 100 
Postsecondary 
State licensing/credentialing exam 4 14 21 3 5 1 48 93 7 100 
National licensing/credentialing exam 3 7 18 4 4 2 48 90 10 100 
Industry developed, employer 
validated exam 0 12 24 7 4 1 48 84 16 100 
Commercially developed exam 6 22 13 6 1 48 68 32 100 
Institutionally developed skill exam 1 1 16 15 4 1 48 65 35 100 
CTE course or program completion 11 7 8 19 2 1 48 58 42 100 
Grade point average 12 4 7 21 3 1 48 52. 48 100 
State developed skill exam 1 3 3 38 1 2 48 16 84 100 


Exhibit reads: Most secondary state directors indicated that at least some of the LEAs in their state used industry-developed, employer- 
validated exams to assess students’ technical skill attainment (43). Thirty-nine postsecondary state directors indicated that at least some 
IHEs used state licensing and credentialing exams to assess postsecondary technical skill attainment. 

NOTE: Percentage calculations exclude “Don’t know” and “No response.” 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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A postsecondary CTE administrator in one case study state advocated the use of grade point 
average (GPA) to document technical skill attainment: “We measure technical skill profi- 
ciencies through multiple demonstrations throughout the semester. Why would I take one 
test and say that one paper-and-pencil test is your level of proficiency? ... We feel a grade of 


C or better in those courses is a strong indicator of technical skill attainment.” 


Among local subgrantees, 82 percent of LEAs and 87 percent of IHEs assessed technical 
skill attainment for at least some of their students by course or program completion, and 56 
percent of LEAs and 63 percent of IHEs assessed all students using this method (Exhibit 
3.20). State licensing and credentialing exams were used more frequently to assess at least 


some postsecondary students (84 percent) than secondary students (44 percent). 
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Exhibit 3.20. 
Percentage of LEAs and IHEs according to the extent to which they used various methods 
to assess students’ technical skill attainment in 2008-09 


BAI B Some ot most @ None Don’t know 


Coutse or program completion 
Locally developed exam 

Grade point average 
State-developed exam 

LEA State licensing/credentialing exam 
National licensing/credentialing exam 
Industry-developed, employer-validated exam 


Commercially developed exam 


Other 


Coutse ot program completion 

State licensing/credentialing exam 
National licensing/credentialing exam 
Institutional developed exam 

IHE Grade point average 
Industry-developed, employer-validated exam 


Commercially-developed exam 


State-developed exam 


Other 


0 20 40 60 80 100 


Percent 


Exhibit reads: Eighty-two percent of LEAs and 87 percent of IHEs used course or program completion to assess at least some students’ 
technical skill attainment. 

! Interpret data with caution. These items have a response rate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. Detail may not sum to totals because of rounding. 

SOURCE: LEA and IHE Surveys, 2009. 


Very few states had a system of state- or commercially developed assessments that would al- 
low them to offer industry-aligned assessments for all their CTE programs (nine secondary, 
four postsecondary) (Exhibit 3.21). About half of secondary state directors reported having a 
state- or commercially developed assessment system that allowed them to cover a subset of 


programs, while only 15 postsecondary state directors reported the same. Several states were 
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in the process of designing or implementing a system (21 secondary, 18 postsecondary) or 


considering their development options (16 secondary, 14 postsecondary). 


Exhibit 3.21. 
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Number of states according to progress made in developing technical skill assessments 


Designing or transitioning to 
TSA system 


Considering development options 


Adopted commercial TSAs for a subset 
of programs 


Have an approved list of TSAs 


Have state-developed TSAs for a subset 
of programs 


No plans to develop a TSA system 


Other 


Have state-developed TSAs for all 
programs 
Adopted commercial TSAs for all 
programs 


No tesponse 


(TSAs), by education level 


Secondary Postsecondary 
a 2 a 1: 
a 1 ae 1+ 
as mm ° 
mm ° a 1 
mm 10 Ma 6 
ms a ° 
m4 ms 
mm m2 
m4 B2 
11 11 


0 10 20 30 


Number of states 


0 10 20 30 


Number of states 


Exhibit reads: Only nine secondary and four postsecondary state directors had state- or commercially-developed assessments that would 
allow them to offer TSAs for all their CTE programs. 


NOTE: N = 51 secondary, 48 postsecondary. 


SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


In one case study state, LEAs had four different ways to measure technical skill attainment 


because, at the time of the site visit, the state was able to assess only 12 of the state’s nearly 


10,000 concentrators using an industry-based certification exam. According to a local pro- 


vider, the state has a specific hierarchy: “The highest one is ... by industry certification. The 


next down is third-party certification. ... The third is whether they had a dual credit course 
for which they received credit. ... The fourth and final option is what we call ‘locally devel- 


oped criteria.” 


Another state chose to assess all concentrators through some means and track the method 


of assessment. As a secondary administrator explained, 
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We felt it was better to look at all concentrators and assess them on 
[whether they met] the gold standard as we defined it in our plan ... or did 
they meet it based on the alternative measure, which is the teachers certify- 
ing that they achieved the competencies based on the curriculum frame- 
works provided to them. That was the only way we were able to look at all 
the concentrators rather than look [only] at the people who took an assess- 


ment. 


Even the one case study state using the Department’s guidance for secondary and postsec- 
ondary technical skill attainment noted limitations in its data: “We don’t report assessment 
data to [the Department] for pathways where there is not an approved statewide assessment. 
That’s just something ... we’re going to have to live with until we get valid assessments in all 
the programs. For the others, we try to push for proof of self-reported results. The validity 


has improved now that we’ve started doing monitoring visits.” 


When to measure attainment 


Cost 


Perkins IV and the nonregulatory guidance allow states flexibility in determining whether to 
assess student attainment at the end of a course, at the end of a program, or at the end of a 
program of study. Some states may choose to evaluate technical skill attainment on a forma- 
tive basis—as students are learnine—while others may assess it on a summative basis—once 


students have completed a CTE program (U.S. Government Accountability Office 2009). 


The type of assessment may determine when students are assessed. For example, assess- 
ments through a third party or through licensing and certification agencies may be designed 
to measure attainment at the end of a program or when students are ready to enter the work- 
force. Other measurement approaches—such as state- or locally developed assessments, 
grades, student portfolios, or proficiency demonstrations—may offer the option of forma- 
tive or summative assessments. If states are assessing students at different points along their 


educational path, the results could limit data comparability. 


States cited cost as an impediment to offering technical skill assessments. According to the 
GAO, the cost of developing statewide assessments or acquiring third-party assessments can 
be prohibitive for some institutions and students who may need to pay for the assessment 
(U.S. Government Accountability Office 2009). 


A director in one state claimed to know local subgrantees that chose not to accept Perkins 
funding, in part due to the requirement for technical skill assessments. According to him, “If 
they want this mountain of data and they want technical assessments, they need to pay for 
[them]. I know some people opted out of even applying for Perkins; they didn’t think it was 


worth it.” In another state, a director explained that using nationally recognized exams may 
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be cost-prohibitive for some LEAs, because they would have to pay the exam fees for their 


students. 


Access to data 
Challenges in accessing data may contribute to states’ use of different types of assessments. 
While states have immediate access to results based on grades, student work, and assess- 
ments taken by students while in school, data for assessments offered through external li- 
censing and certification organizations may not be immediately available because of timing 


and privacy reasons. 


Some licensing boards offer assessments only a few times a year!® or after students have ob- 
tained a threshold amount of work experience;'? therefore, students may not complete the 
assessment process by the December CAR deadline. An administrator in one case study state 
identified the early childhood education field as particularly problematic because students 
“have five years after they get the degree to take the certification exam, and we don’t even 
know if they do it or not.” A local dean noted that “a lot of students take those [exams] after 
they graduate, so we may not be able to get those data or tie them to specific names. We 
could try to collect that afterward, but it would mean [we’d need] a staff person we don’t 


have [now] calling the student.” 


Citing privacy concerns, some licensing and certification organizations may decline to share 
assessment information with anyone other than the student. States report considerable diffi- 
culty in obtaining student-level assessment records from many licensing and certification 
agencies, even from other state agencies, such as state nursing boards (Data Quality Institute 
2006b; U.S. Government Accountability Office 2009). 


One case study state administrator related, “I’ve tried to get [scores on] third-party tests 
[through administrative data matches], and [test administrators] won’t do it; nobody does it.” 
According to another, “That’s a huge challenge. Once students graduate, they’re not required 
to come back to tell their teachers ‘I received my cosmetology license or my RN license’ or 
whatever. So our school districts don’t report on that—or only report on this handful of 
students [who] do come back to. . . show their certificates to their instructors.” Another 
state administrator discussed the challenges in accessing data, “The board of nursing will 
give us that information, but it will cost a lot of money; we will get the information when we 
pay the $2,000 or $3,000.” 


'8 For example, the American Association of Electrodiagnostic Technologists’ (AAET) website indi- 
cates that the organization will offer three examination periods in 2011: March 5-19; June 4-18; and 
October 1-15. 

'9 An example is the Automotive Service Excellence (ASE) certification. To earn an ASE certificate, 
students must pass an ASE test and have at least two years of relevant full-time hands-on work expe- 
rience in the motor vehicle service industry (National Institute for Automotive Service Excellence 
2010). 
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Completion 


The definition of the secondary completion indicator in the nonregulatory guidance asks for 
the proportion of concentrators who earn a high school diploma, pass the GED tests, or ob- 
tain any equivalent diploma offered by a state (Exhibit 3.22). Postsecondary subgrantees re- 


port on students who earn a degtee, certificate, or industry-recognized credential. 


Exhibit 3.22. 
Nonregulatory guidance: Completion 


Secondary 


381: Secondary school completion Numerator: Number of CTE concentrators who earned a regular secondary school di- 


ploma, earned a General Education Development (GED) credential as a State-recognized 
equivalent to a regular high school diploma (if offered by the State) or other State- 
recognized equivalent (including recognized alternative standards for individuals with 
disabilities), or earned a proficiency credential, certificate, or degree, in conjunction with a 
secondary school diploma (if offered by the State) during the reporting year. 
Denominator: Number of CTE concentrators who left secondary education during the 
reporting year. 


Postsecondary 


2P1: Completion of credential, certificate, Numerator: Number of CTE concentrators who received an industry-recognized creden- 


or degree 


tial, a certificate, or a degree during the reporting year. 


Denominator: Number of CTE concentrators who left postsecondary education during 
the reporting year. 


Exhibit reads: The nonregulatory guidance asks for the proportions of concentrators who earn a high school diploma, pass the 
GED tests, or obtain any equivalent diploma offered by a state for secondary completion. Postsecondary subgrantees report on 


students who earn a degtee, certificate, or industry-recognized credential. 
SOURCE: Justesen (2007a). 


State approaches to these indicators appear to vary, possibly due in part to issues such as 
identifying concentrators who leave and access to data. Among secondary state directors, 49 
had at least some confidence in their data on secondary student completion of a high school 
diploma, and 32 were very confident (Exhibit 3.5). Most secondary directors (35) did not 
know or indicated that state-recognized equivalent credentials did not apply to their state, 
and nearly half said the same for GED and proficiency credentials. Twenty states used the 
nonregulatory guidance verbatim for this secondary measure (Exhibit 3.10). 


Postsecondary state directors had confidence in the data reported for this measure as well, 
with 45 indicating at least some confidence and 17 reporting that they were very confident 
(Exhibit 3.7). Nineteen states used the guidance verbatim when developing their postsec- 
ondary measures (Exhibit 3.11). 


Identifying concentrators who leave 


The completion measures consider concentrators who left secondary or postsecondary edu- 


cation during the reporting year; however, determining whether a student has left may pose 
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difficulties for both sectors. As previously discussed with respect to academic attainment, in 
order to identify leavers, local secondary schools and districts must distinguish among stu- 


dents who drop out, transfer to another school, or graduate. 


In the postsecondary sector, in addition to students who graduate after completing a pro- 
gram, some leave without graduating but with adequate skills to obtain employment, and 
others transfer to another two- or four-year institution without earning a degree (Hagedorn, 
Cabrera, and Prather 2010-11). Still more take time off and return later, while some students 
leave and never return. Some students attend only a term or two each year, taking multiple 
years to complete a degree. Students do not have to inform their institution when or why 
they leave, so states must create a clear definition of a leaver and then distinguish those stu- 


dents from others. 


Access to data 
Some states may have difficulty obtaining information about secondary students who take 
GED tests. State secondary agencies may not administer the GED database and may not 
have ready access to the data (U.S. Government Accountability Office 2009; Richards and 
Schoelkopf 2008). According to the GAO, “About one-third of all states cited their ability to 
access accurate GED data as a great or very great challenge” (U.S. Government Accounta- 
bility Office 2009). 


States also may struggle to obtain certification and licensure data for both sectors. As noted 
with respect to technical skill attainment, the timing of some assessments does not align with 
Perkins reporting requirements, and privacy concerns may prevent external organizations 
from sharing information about student assessments with state education agencies or local 
suberantees. If states are limited to assessing students who obtain an external license or cer- 
tificate in the reporting year, the accomplishments of students passing those exams after the 


December CAR deadline will not be captured. 


Some states can collect completion data for postsecondary students earning an external cre- 
dential, such as a nursing certificate awarded by a state nursing board. Few states, however, 
can capture this information for all external credentials, and some do not include any exter- 
nal credentials when calculating postsecondary completion. Although states may be using the 
same written definition—and reporting information about attainment of credentials, certifi- 
cates, and degrees—some states are not capturing as many kinds of completions as other 
states. Therefore, completions may be underestimated in results reported to the Department 


and the comparability of data may be limited. 
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Graduation rate 
Congress added the secondary graduation rate indicator in Perkins IV, indicating that it should 
be aligned with graduation rates that states report for ESEA accountability. The Department 
clarified and refined the indicator definition in its nonregulatory guidance (Exhibit 3.23). 


Exhibit 3.23. 
Nonregulatory guidance: Graduation rate 


Secondary 


4S1: Graduation rate Numerator: Number of CTE concentrators who, in the reporting year, were included as 
graduated in the State’s computation of its graduation rate as described in Section 
1111(b)(2)(C)(vi) of the ESEA. 
Denominator: Number of CTE concentrators who, in the reporting year, were included 
in the State’s computation of its graduation rate as defined in the State’s Consolidated 
Accountability Plan pursuant to Section 1111(b)(2)(C)(vi) of the ESEA. 

Exhibit reads: The Perkins IV secondary graduation rate indicator is aligned with graduation rates that states report for ESEA 


accountability. 
SOURCE: Justesen (2007a). 


As with the academic attainment measure, some states may find it difficult to access data or 
use the methodology for calculating ESA graduation rates when analyzing data for Perkins 
(Richards 2009; Richards and Schoelkopf 2008). Twenty-three states used the nonregulatory 
guidance verbatim for their measure definition, and 27 consulted the guidance (Exhibit 3.10). 
Only one state elected not to use the guidance to develop its definition for this measure. 
Among the six case study states, only one used a methodology different from that in the 


nonregulatory guidance. 


Retention or transfer 
Por Perkins LII, states reported student enrollment in postsecondary education, entrance into 
employment, and enlistment into the military under a single placement indicator. Perkins IV 
retains that same indicator for secondary education but creates two separate indicators for 
the postsecondary level (Sec. 113(b)(2)(B)). 


e Retention in postsecondary education or transfer to a baccalaureate degree pro- 


gram. 


e Placement in employment, the military, or an apprenticeship program. 


The nonregulatory guidance clarifies the legislative language by specifying that “retained” 
students are those who remain enrolled in their original postsecondary institution, while 


“transfers” are students who enroll in a different two- or four-year postsecondary institution 
(Exhibit 3.24). 
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Exhibit 3.24. 
Nonregulatory guidance: Retention or transfer 


Postsecondary 


3P1: Student retention or transfer Numerator: Number of CTE concentrators who remained enrolled in their original post- 
secondary institution or transferred to another 2-or 4-year postsecondary institution dur- 
ing the reporting year and who were enrolled in postsecondary education in the fall of the 
previous reporting year. 
Denominator: Number of CTE concentrators who were enrolled in postsecondary edu- 
cation in the fall of the previous reporting year and who did not earn an industry- 


recognized credential, a certificate, or a degree in the previous reporting year. 


Exhibit reads: The nonregulatory guidance clarifies the legislative language for the postsecondary retention or transfer indicator by 
specifying that “retained” students are those who remain enrolled in their original postsecondary institution, while “transfers” are 
those who enroll in a different two- or four-year postsecondary institution. 

SOURCE: Justesen (2007a). 


Twenty-one states used the nonregulatory guidance verbatim for retention or transfer, and 
only one state did not use the guidance when developing its measure definition (Exhibit 
3.11). Among the six case study states, only one used the nonregulatory guidance as written. 
The other five states made alterations that appeared to go beyond minor wording changes to 
modifications that could result in data that are not comparable across states, although the 


methodology may be valid and reliable within a state. 


For example, one state used most of the text of the nonregulatory guidance but looks at stu- 
dents enrolled in the entire previous year, not just the fall term, resulting in a larger cohort of 
concentrators to evaluate. Another state assesses CTE participants who enroll in other high- 
er education institutions or who become concentrators during the course of the reporting 
year. This means the state is evaluating outcomes of participants rather than concentrators— 


a larger group of students with less exposure to CTE. 


All state directors responding to the survey had at least some confidence in the accuracy of 
the retention or transfer data (Exhibit 3.7). Most state directors had less than a high degree 
of confidence, however, suggesting that questions may exist about how some data are ob- 
tained or reported. Only 27 percent of IHE directors indicated they were very confident in 
the accuracy of data for this measure, although 87 percent had at least some confidence 
(Exhibit 3.8). 


Other research has noted that access to data, identifying concentrator cohorts, and timing 
may impede implementation of the measure (U.S. Government Accountability Office 2009; 
Richards and Schoelkopf 2008). These limitations may contribute to the disparities noted 


among state measure definitions and are explored in more depth in the following sections. 
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Access to data 

Information about students who transfer or persist in the same institution may not be equal- 
ly accessible to all states and local subgrantees. Nearly all local respondents in the case study 
states cited obstacles in tracking students from one education level to the next. According to 
one IHE administrator, “High schools want to know how their kids [do] when they get here, 
and we want to know what happens when they leave us, but we’re still having a difficult time 
tracking them.” Another postsecondary administrator agreed, “It really is the transitional da- 
ta that are the hardest to get. Everybody knows [they’re] necessary. It’s a challenge [to get 


them] at just the local interagency level, and it’s an incredible challenge statewide as well.” 


While matching student records across postsecondary institutions or education levels may 
pose a challenge for some states, obtaining data from out-of-state and private institutions is a 
barrier as well (U.S. Government Accountability Office 2009). The GAO cited cost as a 


challenge for states seeking to track students’ enrollment.” 


Concentrator cohorts 
Unlike other measures, the retention or transfer measure in the nonregulatory guidance 
looks back in time to assess student progress. Rather than exploring what percentage of the 
reporting year concentrator cohort continues in education in the future, the measure deter- 
mines “of those enrolled in the reporting year, what percentage was enrolled in the fall of 
last year.” In addition, comparing the whole of the reporting year to the fall of the prior year 
does not take into account the coursetaking patterns of some postsecondary students, who 
may enroll part time or stop out and re-enroll in different terms during the year. A 2001 re- 
port found that 30 percent of the students included in the study’s sample “stopped out,” 
meaning they enrolled, did not enroll, and then enrolled again across a sequence of terms 
(Stratton, O’Toole, and Wetzel 2001). 


Among the case study states, only two used a definition that looked at the current concentra- 
tor cohort enrollment sometime in the prior year. One had not yet finalized its measure at 
the time of the case study visits. The other three states appeared to evaluate the future reten- 
tion or transfer outcomes of the reporting year concentrators. In one state, an administrator 


explained, 


°° For example, states and IHEs that wish to acquire enrollment data from out-of-state and private in- 
stitutions may rely on the National Student Clearinghouse (NSC), which charges a fee for member- 
ship. The NSC offers three options for membership in the NSC’s StadentTracker service, which 
provides reports on the enrollment status of prospective, current, and former students: (1) pay an an- 
nual fee equal to enrollment times $.10, with a minimum annual fee of $300; (2) pay an annual fee of 
enrollment times $.05 ($150 minimum) by participating in two other free NSC services (DegreeVerify 
and EnrollmentV erify) or reporting additional data elements being added to the NSC database; or 

(3) participate in the two free NSC services, report the additional data elements, and enroll in S7w- 
dentTracker for free (National Student Clearinghouse 2010). 
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I don’t like persistence because of the part-time status of the community 
college students. I followed one cohort. Were they there next year? No. 
They were in my leaver cohort. I had 80 percent of the students come back 
eventually over a three- or four-year period. Measuring persistence the way 
we're doing it, year-to-year persistence, is kind of a remnant from the four- 
year degree model. The median time to a two-year degree is seven semes- 


ters—and that’s not contiguous semesters. 


An administrator in one case study state mentioned that the state might change its definition 
for retention or transfer to follow future student outcomes and make the definition more 
useful for internal reporting. He advocated looking at how the concentrator cohort in the 
current year fared in terms of placement in the next fall term: “We could report [on the co- 
hort] in a consistent way and still get at the same figures if we waited and looked at the trans- 
fer rate in September. For the 2009-10 cohort, you would look at the transfer rate in 
September 2010.” 


Evaluating student retention or transfer outcomes in this way could pose other challenges, 


however. For example: 


e Numerator: Number of concentrators in the reporting year who did not earn a 
degree, certificate, or credential and who remained enrolled in their original 
postsecondary institution or transferred to another two- or four-year postsec- 


ondary institution at any time in the following reporting year. 


e Denominator: Number of concentrators in the reporting year who did not 


earn a degree, certificate, or credential. 


This sample approach first aligns the numerator and denominator by specifying that both in- 
clude “concentrators who did not earn a degree, certificate, or credential.” This approach al- 
so looks forward, rather than backward, in evaluating student outcomes. The method, 
however, raises an issue about when data are reported. In this example, if states identify a 
group of concentrators in a given reporting year, they must wait until the following reporting 
year has ended to determine if students persisted or transferred (Exhibit 3.25). Further, it 
may take time to obtain the data once the following year has ended. A state identifying its 
concentrator group in the 2009-10 academic year would have to wait to see if those students 
attended a postsecondary institution during the 2010-11 reporting year. Depending upon its 
source, the data may become available for analysis in summer or fall of 2011, which would 
allow results to be reported for the 2011 CAR. 
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Exhibit 3.25. 
Timeline of Perkins data availability for placement indicator 
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Exhibit reads: A state identifying its concentrator group in the 2009-10 academic year would determine which of those students 
attended a postsecondary institution during the 2010-11 reporting year. Depending upon their source, the data may become available 
for analysis in summer or fall 2011, and the state could report results for the 2011 CAR. 


Reporting timelines 
The issue of when to report performance results is not exclusive to the retention or transfer 
measure. Several other secondary and postsecondary measures pose challenges for states try- 
ing to obtain and report data six months after the end of the program year, including sec- 
ondary and postsecondary technical skill attainment and placement and postsecondary 
completion. For all these measures, states must obtain data—such as subsequent enrollment, 


employment, and licensure or certification results—that may not be available in time to meet 
the CAR deadline. 


States address the timing issue in a variety of ways, leading to even less comparability in the 
data reported to the Department. Some states may vary the year reported on the CAR by in- 
dividual measure. For example, for the 2010 CAR submission, a state might report 2009-10 
concentrator outcomes for a few measures and 2008-09 concentrator outcomes for the rest. 
Others may use the prior reporting year concentrator cohort for all measures. In that case, 
the state reports outcomes for 2008-09 concentrators for all measures in the December 
2010 CAR submission. Four of the six case study states waited a year to report placement 


outcomes for postsecondary students, and five did the same for secondary placement. 
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Placement 
Perkins IV asks states to report secondary student placement in postsecondary education or 
training, employment, or the military, and postsecondary student placement in military ser- 
vice, an apprenticeship program, or employment. The postsecondary placement indicator 
does not include enrollment or retention in postsecondary education, which are now ad- 
dressed by the retention or transfer indicator. It includes a reference to high-skill, high-wage, 


ot high-demand occupations that does not appear in the secondary placement indicator. 


The Department’s nonregulatory guidance clarifies the timeline for assessing student place- 
ment, indicating that states should evaluate student placement outcomes in the second quar- 
ter after the end of the program year (Exhibit 3.26). It does not, however, retain mention of 


high-skill, high-wage, or high-demand occupations for the postsecondary placement meas- 


ure. 
Exhibit 3.26. 
Nontregulatory guidance: Placement 

Secondary 

5S1: Placement Numerator: Number of CTE concentrators who left secondary education and were 
placed in postsecondary education or advanced training, in the military service, or em- 
ployment in the second quarter following the program year in which they left secondary 
education (i.e., unduplicated placement status for CTE concentrators who graduated by 
June 30, 2007 would be assessed between October 1, 2007 and December 31, 2007). 
Denominator: Number of CTE concentrators who left secondary education during the 
reporting year. 

Postsecondary 

4P1: Placement Numerator: Number of CTE concentrators who were placed or retained in employment, 


or placed in military service or apprenticeship programs in the 2nd quarter following the 
program year in which they left postsecondary education (i.e., unduplicated placement 
status for CTE concentrators who graduated by June 30, 2007 would be assessed between 
October 1, 2007 and December 31, 2007). 


Denominator: Number of CTE concentrators who left postsecondary education during 
the reporting year. 


Exhibit reads: The Department’s nonregulatory guidance indicates that states should evaluate student placement outcomes in the 
second quarter after the end of the program year. 
SOURCE: Justesen (2007a). 


Most secondary state directors had some confidence in the accuracy of local data reported 
for the placement measure, but few were very confident about postsecondary education and 
training data (8 states), military enlistment (9 states), and employment (10 states) (Exhibit 
3.5). A small number of postsecondary directors had no confidence in the accuracy of mili- 


tary enlistment (6) and apprenticeship data (8) (Exhibit 3.7). 
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The six case study states each defined this measure differently for both the secondary and 
postsecondary levels, and none used the nonregulatory guidance as written. For example, 
one state evaluated placement outcomes for concentrators from the prior year rather than 
from the reporting year, which means it assessed placement outcomes for a different cohort 
of concentrators than those evaluated in some other performance measures. Another state 
included enrollment in postsecondary education or training as part of its postsecondary 
placement measure, even though those outcomes also were captured as part of the retention 


or transfer indicator. 


The survey of state directors asked how access to data, cost, and other issues affected their 
ability to report placement information accurately. Among secondary state directors, 42 ex- 
pressed at least some concern about the effect of response rates on the accuracy of results 
(Exhibit 3.27). Many also reported concerns about access to Social Security numbers (SSNs) 
(40) and the cost of following students who left secondary education (39). Only 11 second- 
ary state directors indicated that the cost of record matching with other agencies had a great 
impact on the state’s ability to report placement results, while 10 reported it had no impact. 
Forty-three postsecondary state directors indicated that collecting out-of-state employment 
data had some impact on the state’s ability to report placement data, and 22 said it had a 
great impact (Exhibit 3.28). Additionally, most reported that collecting self-employment data 


had at least some impact on their ability to report placement outcomes. 
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Exhibit 3.27. 


Number and percentage of secondary state directors according to the impact of the 


following issues on their state’s ability to report placement data, by issue 


(5—point scale from No impact to Great impact) 


Number 
Mote than 
Gteat noimpact, No At least 
impact lessthan impact Don’t No some 
(5) = gteat (2-4) = (1) know response Total | impact 
Cost 
Cost of follow-up studies of 
students who left 19 20 6 4 2 51 80 
Cost of record matching 
with other agencies 11 23 10 6 1 51 68 
General 
Getting an adequate 
response rate 15 27 3 4 2 51 86 
Access to SSNs 33 7 5 5 1 51 80 
Access to education data 
Collecting in-state 2-year 
public postsecondary data 12 26 8 4 1 51 76 
Collecting in-state 4-year 
public postsecondary data 16 22 6 6 1 51 76 
Collecting in-state private 
postsecondary data 15 18 6 11 1 51 66 
Collecting out-of-state public 
or private postsecondary 
data 21 16 3 10 1 51 74 
Access to employment data 
Collecting self-employment 
data 17 19 2 12 1 51 72 


Collecting out-of-state 


employment data 


132 
Percent 
No Don’t 
impact know Total 
12 8 100 
20 12 100 
6 8 100 
10 10 100 
16 8 100 
12 12 100 
12 22 100 
6 20 100 
4 24 100 


Exhibit reads: Among secondary state ditectors, 42 expressed at least some concern about the effect of response rates on the accuracy of 
results. Many also reported concerns about access to Social Security numbers (SSNs) (40), the cost of following students who left 


secondary education (39), and collecting out-of-state employment data (39). 
NOTE Percent calculations exclude “No response.” 
SOURCE: Secondary State Director Survey, 2009. 
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Exhibit 3.28. 
Number and percentage of postsecondary state directors according to the impact of the 
following issues on their state’s ability to report placement data, by issue 


(5—point scale from No impact to Great impact) 


Number Percent 
Mote than 
no impact, 
Great less than No At least 
impact great impact Don’t No some No Don’t 
(5) (2-4) (ad) know tesponse Total | impact impact know Total 
Cost 
Cost of follow-up studies 
of students who left 14 19 7 6 2 48 72 15 13 100 
Cost of record matching 
with other agencies 9 23 11 4 1 48 68 23 9 100 
General 
Getting an adequate 
response rate 12 19 8 6 3 48 69 18 13 100 
Access to SSNs 22 10 11 4 1 48 68 23 9 100 
Access to education data 
Collecting in-state 2-year 
public postsecondary data 11 16 17 2 2 48 59 37 4 100 
Collecting in-state 4-year 
public postsecondary data 13 14 17 3 1 48 57 36 6 100 
Collecting in-state private 
postsecondary data 14 17 12 4 1 48 66 26 9 100 
Collecting out-of-state public 
or ptivate postsecondary 
data 21 17 6 3 1 48 81 13 6 100 
Access to employment data 
Collecting self-employment 
data 14 25 1 7 1 48 83 2 15 100 


Collecting out-of-state 


employment data 
Exhibit reads: Forty-three postsecondary state directors indicated that collecting out-of-state employment data had some impact on the 
state’s ability to report placement data, and 22 said it had a great impact. Most reported that collecting self-employment data had at least 
some impact on their ability to report placement outcomes (39). 
NOTE: Percent calculations exclude “No response.” 
SOURCE: Postsecondary State Director Survey, 2009. 
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States varied considerably in how they defined their measures, which may make it difficult 
for the Department to report comparable national data. Study findings indicate that differ- 
ences in student cohorts, difficulties and costs in accessing necessary data, variation in track- 
ing methods, and inconsistent reporting timelines may diminish data comparability for this 


measure. These issues are discussed in more depth below. 


Concentrator cohort 
The nonregulatory guidance asks states to track placement outcomes for all concentrators 
who leave secondary or postsecondary education. As a result, the cohort evaluated in this 
measure includes concentrators who leave without graduating or earning a credential and 
those who never complete a CTE program, in addition to concentrators who do complete 


their programs and graduate or earn a postsecondary credential. 


If the intent of the measure is to see if CTE students meeting the concentrator threshold 
continue in their education or find employment, this measure likely answers that question. If 
the intent is to determine if students learned the academic and technical skills necessary to 
obtain employment or continue their education, this measure may not offer the desired in- 
formation, because it also assesses outcomes for students who did not complete a program 


or graduate and students who may have taken only a few courses. 


States vary in how they interpret and apply the nonregulatory guidance when identifying the 
concentrators assessed in this measure. Some states include all concentrators who leave, 
while others include only those who complete a CTE program or earn a diploma or creden- 
tial. For example, at the postsecondary level, two of the six case study states evaluated only 
graduates, rather than all concentrators who left. At the secondary level, two states included 
only high school graduates, and two others included only program completers in the meas- 
ure. An administrator in one state explained, “It didn’t seem appropriate to hold the state ac- 
countable for the performance [and placement] of people who had not completed the entire 


program.” 


Placement period 
Students may leave their schools or institutions at any time during an academic year, not 
necessarily when the school year ends. By identifying the second quarter after the end of the 
program year as the period when placement is assessed, the measure combines outcomes for 
concentrators who have had different amounts of time to find and start a job. For example, 
a student who attends only in the fall term may have from January to October to be hired 
and begin work. A student who leaves a high school or postsecondary institution at the end 


of the spring term has less time (June to October) to do the same. 


This study found variation among the placement periods used by the six case study states. 


For secondary education, two states considered secondary placement outcomes six months 
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after high school exit, while another looked at placement in the third quarter after exit. For 
postsecondary, one state reviewed all four quarters after the end of the year, and another 


looked at the third quarter after exit, rather than the second quarter after the end of the year. 


The state that evaluated all four quarters questioned the wisdom of limiting the time frame for 
measurement of placement to a single quarter. An administrator said, “Given the instability of 
the employment data when we looked at the use of a single specific quarter—especially either 
October through December [as suggested in the nonregulatory guidance] or January through 
March, which are both high and low quarters for retail, agricultural, construction, and a variety 


of other occupational areas—we decided to use a mote stable indicator.” 


Tracking methods 
States employ two primary methods for tracking student placement outcomes: student sur- 
veys and administrative record matching. Some states obtain all their placement data through 
one method or the other, although some use both approaches (1.e., record matching for 


postsecondary enrollment and surveys for employment outcomes). 


The majority of measure definitions from the six case study states did not specify the meth- 
od used to collect placement data. The secondary measures for two states, however, indicat- 
ed that one used surveys while another used both surveys and administrative record 
matching to collect secondary placement data. According to the GAO, 38 states used a sur- 
vey administered by the state, school district, or a third party to track secondary placement 
outcomes, and 41 states used Unemployment Insurance (UI) wage record data (U.S. Gov- 


ernment Accountability Office 2009). 


SURVEYS 

According to a report prepared for the Department, states use a wide variety of survey 
methods to collect data (Klein et al. 2006). Some conduct the survey by mail, phone, or 
online, and others ask local subgrantees to administer the surveys. In states where local pro- 
viders collect the data, some states may develop standard surveys that their LEAs and IHEs 
implement, while others allow flexibility in the methodology. For example, local providers 
might have the option to call former students individually, administer a mail or telephone 
survey, talk with a student’s family or friends, or conduct an exit interview before students 


leave. 


Four of the case study states indicated they asked LEAs to collect secondary placement data. 
One asked local staff to complete a post-school survey that includes information about 
graduates who enter postsecondary education, employment, or military service. This state al- 


so asked each local provider to identify how they obtained the placement information. 
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Surveys may pose resource and validity challenges to some states. It can be expensive to de- 
velop, administer, and analyze telephone and mail surveys, and although online surveys may 
be less expensive, states or local subgrantees must still design, program, and monitor online 
sutveys (RTI International and Academy for Educational Development 2003). Reaching stu- 
dents is an ongoing issue that can affect response rates, because their permanent addresses 
and phone numbers may change. Depending on whether a state or local provider collects 
student or parent e-mail addresses, there may be constraints in connecting students to online 
surveys. In the study surveys, 42 secondary state directors and 31 postsecondary state direc- 
tors indicated that getting an adequate response rate had at least some impact on their ability 
to report placement data (Exhibit 3.27 and Exhibit 3.28). 


One local administrator described his new approach to capturing placement data after gener- 
ating a low response rate—about 30 percent—using mail surveys: “When they come back to 
get their senior yearbooks, my assistant sits there with the follow-up surveys. ... [With] the 
yearbooks, we get about 95 percent. We also go into the senior English classes to ask them 


about the students who graduated last year.” 


In another state an IHE instructor said “it’s pretty easy” to obtain placement information for 
former students: “We’ve got a small program, and students notify us or employers notify 
us.” Another IHE administrator collaborates with the university system to track students in- 
to local baccalaureate institutions. She said, “We know that because of a local arrangement, 


not because there is a statewide system that provides that information.” 


ADMINISTRATIVE RECORD MATCHING 

Matching student records with administrative enrollment and employment records main- 
tained by other agencies is another way to follow students into further education and the 
workforce. Record matching can eliminate some of the costs associated with developing and 
implementing complex structures to collect survey data. Record matching also may be con- 
ducted mote easily at the state level, if the capacity exists to do so, removing some of the 


burden on local subgrantees. 


Some states will, however, need to pay other state or national agencies to provide records or 
perform matches, which can be costly.2! According to survey results, 34 secondary and 32 
postsecondary state directors said that costs for matching administrative records with other 
agencies had at least some impact on their reporting of placement data (Exhibit 3.27 and 
Exhibit 3.28). 


Some states also face legal obstacles in obtaining the data required for the placement meas- 


ure. Access to SSNs allows administrative record matching for employment and military en- 


21 'The discussion on retention or transfer (see p. 126) includes information about the cost of accessing 
postsecondary data through the National Student Clearinghouse, for example. 
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listment, eliminating the self-report bias introduced by surveys. State legal interpretations, 
however, may prohibit some states from collecting SSNs or using them for this type of rec- 
ord matching, Although the survey did not ask states whether they had access to SSNs, it 
asked how access to SSNs affected their ability to report placement data. Thirty-three sec- 
ondary and 22 postsecondary directors believed it had a great impact, and 40 secondary and 
32 postsecondary directors believed it had at least some impact (Exhibit 3.27 and Exhibit 
3.28). 


Even states that have access to SSNs and are permitted to perform administrative record 
matching may find that SSNs are not a perfect student identifier. For example, a single SSN 
might be shared by several people; some people may prefer not to provide their SSNs; and 
institutions may not enter the information correctly. The data available through administra- 
tive records also may not offer all the information that can be captured through a survey. A 
teacher in one case study state shared a concern about administrative record matching: “We 
used to do call-ins three years after [the students] graduated. Now it’s all driven off their So- 
cial Security numbers. We really don’t get the same kind of data we used to get when we 


[did] individual call-ins to talk to students three years later to see what they’re doing.” 


There are several national and state sources for administrative record matching, although 
some states may be prevented from using one or more of them by legal or financial con- 


straints. 


e Unemployment Insurance (UI) wage records: State UI wage record systems 
provide employment and wage information about privately employed individu- 
als and those employed by state and local governments. UI systems do not in- 
clude out-of-state employment, the self-employed, federal employees, workers 
paid solely by commission, and employees of small agricultural employers. 


Matches are conducted using SSNs. 


e Federal Employment Data Exchange System (FEDES): FEDES contains 
records for those who enlist in the military or are employed in many branches 
of the federal government. FEDES uses information from the Office of Per- 
sonnel Management; the Department of Defense, Defense Manpower Data 
Center; and the U.S. Postal Service (USPS) (The Jacob France Institute 2009). 
Matches are conducted using SSNs. 


e National Student Clearinghouse (NSC): The NSC provides information 
about student enrollment in more than 3,300 U.S. postsecondary institutions— 
including private, public, nonprofit, and for-profit—and covers 92 percent of 
USS. college students. Not all postsecondary institutions, and therefore students, 
are included in the NSC database. The NSC performs matches for secondary 


and postsecondary students using a matching key that includes elements such as 
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name, high school attended (for secondary), and birth date or graduation date 
(National Student Clearinghouse 2010). 


e State education databases: Some states can match student records across K— 
12, two-year institutions, and four-year institutions within their states to track 
student enrollment. These databases do not, however, offer information about 
students who attend postsecondary institutions out of state. States may use a 


matching key such as that used by the NSC, an SSN, or a state student identifier. 


Access to administrative records data 
States with access to SSNs still may face hurdles in obtaining some types of administrative 
record data, particularly UI wage record and FEDES data. The Family Educational Rights and 
Privacy Act (FERPA) limits how education data may be used, and the Perkins requirement to 
follow students across time and circumstance is at odds with current FERPA interpretation 
and implementation.” State education agencies may share student data among themselves, 
but cannot relinquish direct control of the data, preventing them from sharing data with oth- 
et noneducation state agencies (Hansen 2003). A few states have surmounted this obstacle 
by obtaining student permission to share their SSNs, by hiring a subcontractor—under di- 
rect control of the education agency—to match the education and employment data, or by 
bringing employment data into the education agency to do the record matching. One sec- 
ondary administrator described engaging a contractor to perform administrative record 


matching: 


We contract placement out with a state university, and we’re going to con- 
tinue to use a contractor until we get our statewide longitudinal data all 
worked up. ... They do UI wage matches for us, FEDES matches, military 
wages. They go through the National Student Clearinghouse to find our 
students at universities and colleges throughout the nation through SAT 


matches and ACT matches as well. 


More than half of secondary and postsecondary state directors reported that their states did 
not use FEDES or the National Student Clearinghouse to acquire placement information 
about students (Exhibit 3.29). Legal restrictions in 12 states prevent secondary state directors 
from accessing UI wage records. More than half of states used state UI wage records (33) 


and state higher education databases (28) to obtain placement results for postsecondary stu- 


2 As of the time of this writing, the Department had issued a Notice of Proposed Rule Making 
(NPRM) under FERPA. According to an April 7, 2011, Department press release (U.S. Department 
of Education 2011), the proposed changes would “safeguard student privacy while clarifying that 
states have the flexibility to share school data that are necessary to judge the effectiveness of govern- 
ment investments in education. Over time, interpretations of FERPA have complicated valid and 
necessary disclosures of student information without increasing privacy protections and, in some cas- 
es, dramatically decreased the protections afforded students.” 
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dents. Only three secondary and eight postsecondary state directors reported using all four 
types of administrative databases, while 15 secondary and six postsecondary state directors 


reported using none of the four databases. 


Exhibit 3.29. 
Number of states according to their use of administrative record matching resources to 
collect placement data, by education level 


MInuse by state MLegal restrictions prohibit use MHNotinuse (CNo response 


State higher education database 


Secondaty State UI wage records 
National Student Clearinghouse 
FEDES 
Number of states 
State UI wage records 
Postsecondaty State higher education database 


National Student Clearinghouse 


FEDES 


0 10 20 30 40 48 


Number of states 


Exhibit reads: Less than half of secondary and postsecondary state directors reported using FEDES (11 secondary, 14 postsecondary) or 
the National Student Clearinghouse (14 secondary, 23 postsecondary) to acquire placement information about their students. Legal 
resttictions in 12 states prevent secondary state directors from accessing UI wage records. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


EMPLOYMENT DATA 

Only 17 states used UI wage records to collect placement data for secondary students, and 
only 11 used FEDES data. Many states (33) used UI wage records to obtain placement data 
for postsecondary students, although only 14 used FEDES data. 


UI wage records do not offer information about self-employed individuals, and 36 secondary 
and 39 postsecondary state directors reported that access to self-employment data had at 
least some impact on their ability to report placement data (Exhibit 3.27 and Exhibit 3.28). 


State directors reported some uncertainty regarding these data, with 12 secondary and 7 
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postsecondary directors indicating that they did not know if access to these data affected 


their ability to report placement outcomes. 


In addition, states with access to UI wage records and FEDES may not have access to indi- 
vidual employment information outside their state borders. UI data provides information 
about many employees within a state, but not across state lines. The majority of secondary 
state directors (39) noted that access to out-of-state employment data had at least some im- 
pact on their state’s ability to report placement data, and 43 postsecondary state directors 
said the same (Exhibit 3.27 and Exhibit 3.28). 


Two local case study respondents expressed concern about placement data derived from 
administrative records, noting that data from employment databases cannot determine if a 
former student is employed in a job related to his or her educational program. As one ex- 
plained, “Some of the larger employers . . . have divisions that would cross many, many dif- 


ferent occupations and many different training programs.” 


POSTSECONDARY ENROLLMENT DATA 

Twenty-one secondary and 28 postsecondary state directors reported having the capacity to 
match students across education levels in their own state. When performing administrative 
record matches, finding students enrolled in postsecondary education and training programs 
outside the state requires matching student records with those in other states or the NSC. 
Only 14 states used NSC data to follow secondary students, in contrast to 23 states that used 
NSC data to track postsecondary students (Exhibit 3.29). 


Secondary state directors registered concern about the impact of access to postsecondary en- 
rollment data on reporting the placement indicator. Only three said that access to out-of- 
state enrollment data had no impact on reporting placement, while 21 said it had a great im- 
pact (Exhibit 3.27). Most responded that collecting enrollment data for both two- and four- 
year institutions had at least some impact, and nearly as many reported the same about col- 
lecting information about students enrolled at in-state private postsecondary institutions. 
Several secondary state directors did not know if collecting out-of-state (10) and in-state pri- 


vate (11) enrollment data would have an impact on their ability to report placement data. 


Fewer postsecondary than secondary state directors reported that access to in-state two- and 
four-year public institution data had an impact on their ability to report placement data 
(Exhibit 3.28). A similar number, however, reported a perceived impact on their ability to 
report placement data due to access to out-of-state (38) and in-state private (31) postsecond- 


aty enrollment information. 


23'The Wage Record Interchange System (WRIS) provides out-of-state private employment data, but 
WRIS data cannot currently be used in Perkins reporting. 
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Reporting timeline 
Information about student placement outcomes may not be available to states or local sub- 
grantees in time for the December 31 CAR submission deadline. For example, some post- 
secondary institutions have fall terms that end in December or January and, therefore, 
obtaining enrollment data from these institutions may not be possible until after December 
31. To capture an entire term of fall enrollment, the NSC suggests making a request the first 
week of January.** As a result, states may be unable to obtain the data they need in time to 
meet the December CAR deadline. Further, if a state uses UI wage records to identify stu- 
dents employed in the second quarter after exiting their postsecondary institution (October 1 
to December 31), the second quarter ends on the same day that the CAR submissions are 
due. Similarly, states using surveys to collect these data also must wait until after December 


31 to ask students if they were employed during the second quarter. 
Nontraditional participation and completion 


The nonregulatory guidance outlines two separate measures for the nontraditional participa- 


tion and completion indicator (Exhibit 3.30). 


4 Personal communication with LaTonya Page, National Student Clearinghouse, August 20, 2010. 
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Exhibit 3.30. 


Nonregulatory guidance: Nontraditional participation and completion 


Secondary 


6S1: Nontraditional participation 


6S2: Nontraditional completion 


Postsecondary 


Numerator: Number of CTE participants from underrepresented gender groups who 
participated in a program that leads to employment in nontraditional fields during the 
reporting year. 

Denominator: Number of CTE participants who participated in a program that leads to 
employment in nontraditional fields during the reporting year. 

Numerator: Number of CTE concentrators from underrepresented gender groups who 
completed a program that leads to employment in nontraditional fields during the report- 
ing year. 

Denominator: Number of CTE concentrators who completed a program that leads to 
employment in nontraditional fields during the reporting year. 


5P1: Nontraditional participation 


5P2: Nontraditional completion 


Numerator: Number of CTE participants from underrepresented gender groups who 
participated in a program that leads to employment in nontraditional fields during the 
reporting year. 

Denominator: Number of CTE participants who participated in a program that leads to 
employment in nontraditional fields during the reporting year. 


Numerator: Number of CTE concentrators from underrepresented gender groups who 
completed a program that leads to employment in nontraditional fields during the report- 
ing year. 

Denominator: Number of CTE concentrators who completed a program that leads to 
employment in nontraditional fields during the reporting year. 


Exhibit reads: The Department’s nonregulatory guidance outlines two separate measures for the nontraditional participation and 


completion indicator. 
SOURCE: Justesen (2007a). 


These measures evaluate students’ involvement in programs leading to occupations not con- 


sidered traditional for their gender (e.g., men in early childhood education programs). The 


6S1 and 5P1 measures of nontraditional participation are the only measures that assess the 


outcomes of participants rather than concentrators. States used the nonregulatory guidance 


verbatim more often for the postsecondary nontraditional indicators than for any other 


postsecondary indicators (Exhibit 3.11), while less than half of states used the guidance ver- 
batim at the secondary level (Exhibit 3.10). 


The number of students reported for these measures is very small in some states, raising 


concerns about states’ and local subgrantees’ ability to meet performance goals. According to 


a secondary administrator, 


One that I really question is nontraditional participation, particularly for the 


smaller agencies. It’s possible that they don’t have any nontraditional pro- 


grams. By law, we can’t require them to operate programs unless we can 


pay for them, which we can’t do. So they’re kind of destined to be sanc- 
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tioned down the road because of something that they can’t do, and they 


don’t have the resources to do. 


According to another administrator, the small populations assessed for the nontraditional 


indicators increase the potential for the misrepresentation of performance. 


Of the 51 secondary state directors who responded, most indicated at least some confidence 
in the accuracy of data for nontraditional participation and completion (48 each) submitted 
by local providers (Exhibit 3.5). Most were less than very confident, however. Two postsec- 
ondary state directors reported having no confidence in the data reported for nontraditional 
participation, as did three for nontraditional completion (Exhibit 3.7). Most postsecondary 
state directors, however, had at least some confidence in the data. About 80 percent of LEA 
directors and more than 80 percent of IHE directors had at least some confidence in the ac- 
curacy of their nontraditional participation and completion data (Exhibit 3.6 and Exhibit 
3.8). 


Tech Prep 


Tech Prep programs offer students a sequential course of study that includes at least two years 
of secondary education and two years of postsecondary education. Perkins III required states 
to report on Tech Prep students as a separate population for each performance measure, but 


Perkins IV introduces a new, separate set of performance indicators for Tech Prep programs. 


Twenty-seven states reported merging their basic and Tech Prep grants as of the 2009-10 
program year; these states are exempt from reporting results for the new indicators. States 
that did not merge funding are responsible for reporting the number of secondary and post- 
secondary Tech Prep students they serve, as well as performance results for the following 


nine Tech Prep indicators: 

e Secondary students 
o Enrolling in postsecondary education 
o Enrolling in postsecondary education in the same field 
o Obtaining a certificate or license 
o Earning postsecondary credit 
o Enrolling in remedial courses in postsecondary 

e Postsecondary students 
oO Being placed in related employment 


o Obtaining a certificate or license 
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o Completing a degree or certificate within normal time 


o Completing a baccalaureate degree within normal time 


States had several reasons for merging their grants, including the philosophy that Tech Prep 


programs are similar to the new programs of study (POS) concept introduced in Perkins IV 


(18 secondary, 15 postsecondary) (Exhibit 3.31). Some wanted to incorporate the features of 


Tech Prep programs into all their state’s CTE programs (23 secondary, 18 postsecondary) 


and some had difficulty identifying Tech Prep students or wanted to avoid the data burden 


associated with the new Tech Prep accountability requirements. None of the states that re- 


tained separate grants had plans to merge them by 2011-12. 


Exhibit 3.31. 


Number of states according to various reasons for merging Tech Prep and basic grant 


Desire to incorporate Tech Prep into 
all CTE programs 


Due to similarity between Tech Prep 
and Programs of Study 


Avoid data burden associated with 
collecting new Tech Prep measures 


Difficulty collecting data on 
Tech Prep students 


Difficulty identifying Tech Prep students 
from other CTE students 


Other 


No response 


Exhibit reads: States reported several reasons for merging their grants, including their philosophy that Tech Prep programs are similar to 
Programs of Study (POS) (18 secondary, 15 postsecondary) and their desire to incorporate the features of Tech Prep programs into all their 


funds, by education level 


Secondary 


Postsecondary 


2; 
a 


a 
a 
ee 
a 
| sa 
2 

2 


Number of states 


state’s CTE programs (23 secondary, 18 postsecondary). 
NOTE: N = 27 secondary, 27 postsecondary. Exhibit includes responses only from states that reported merging Tech Prep and Basic 


Grant funding. Excludes responses from territories and outlying areas. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Number of states 
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In states that merged funding, state directors overwhelmingly reported that the new Tech 


Prep measures imposed a substantial burden (Exhibit 3.32 and Exhibit 3.33). State directors 


also reported that the new measures would support state and local improvement efforts for 


both secondary and postsecondary education. 


Exhibit 3.32. 


Number and percentage of secondary state directors according to the extent to which they 


agreed or disagreed with potential effects of new Tech Prep accountability measures 


Number Percent 


Dis- Dis- 
Agtee agree or Agtee agree or 
or strongly or strongly 
strongly No dis- Don’t No tre- strongly No dis- Don’t 

Potential effect agree opinion agree know sponse Total | agree opinion agree know Total 
Cause state to merge 

Title I and Title II 

funds 3 4 11 5 4 27 13 17 48 22 00 
Able to collect accurate 

data on measures 5 2 14 2 4 27 22 9 61 9 00 
Would like ED to issue 

nonregulatory guidance 7 9 5 2 4 27 30 39 22. 9 100 
Support local program 

improvement efforts 11 4 6 2 4 27 48 17 26 9 00 
Support state program 

improvement efforts 14 3 4 2 4 27 61 13 17 9 100 
Impose a substantial data 

burden on the state 17 1 3 2 4 27 74 4 13 9 00 
Exhibit reads: Among states that merged Tech Prep and basic grant funds, 17 secondary state directors reported that the new Tech Prep 
measures imposed a substantial burden. 


NOTE: Exhibit includes responses only from states that reported merging Tech Prep and Basic Grant funding. Excludes responses ftom 


territories and outlying areas. Percentage calculations exclude “No response.” 
SOURCE: Secondary State Director Survey, 2009. 
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Number and percentage of postsecondary state directors according to the extent to which 


they agreed or disagreed with potential effects of new Tech Prep accountability measures 


Potential effect 

Cause state to merge 
Title I and Title II 
funds 

Able to collect accurate 
data on measutes 

Would like ED to issue 
nonregulatory guidance 

Support local program 
improvement efforts 

Support state program 
improvement efforts 

Impose a substantial data 


burden on the state 


Agtee 
or 
strongly No 


agree opinion agree 


1 5 
5 3 
5 9 

14 4 

14 3 

12 3 


Number 
Dis- 
agree or 
strongly 
dis- Don’t No re- 
know sponse 
11 2 8 
10 1 8 
4 1 8 
1 0 8 
2 0 8 
4 0 8 


Total 


27 


27 


27 


27 


27 


27 


Agtee 
or 


strongly No 


agree 


26 


26 


74 


74 


63 


opinion agree 


26 


16 


47 


21 


16 


16 


Percent 
Dis- 
agree or 
strongly 
dis- Don’t 
know 
58 11 
53 5 
21 5 
5 0 
11 0 
21 0 


Total 


00 


00 


00 


00 


00 


00 


Exhibit reads: Among states that merged Tech Prep and basic grant funds, most postsecondaty state directors believed that the Tech 
Prep measures would support state and local improvement efforts (14 each). 


NOTE: Exhibit includes responses only from states that reported merging Tech Prep and basic grant funds. Excludes responses from 


territories and outlying areas. Percentage calculations exclude “No response.” 


SOURCE: Postsecondary State Director Sutvey, 2009. 


Local directors reported mixed feelings about data quality and the advantages of the new 


measures. Most did not agree that the advantages of the measures outweighed the disad- 


vatitages—more than one-quarter had no opinion on this topic—but 56 percent of LEA di- 


rectors and 58 percent of [HE directors reported that the measures would support local 


improvement efforts (Exhibit 3.34). More than half of secondary and postsecondary local di- 
rectors agreed that the new measures would create a substantial burden. Many indicated they 
could collect data on all measures (53 percent of secondary directors and 45 percent of post- 


secondary directors), but only 38 percent of local directors thought the data collected would 


be of high quality. 
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Exhibit 3.34. 
Percentage of local directors according to the extent to which they agreed or disagreed with 
the potential effects of new Tech Prep accountability measures, by education level 


Somewhat Somewhat 
or strongly No or strongly Don’t 
Potential effect agree opinion disagree know Total 
LEA 
Impose a substantial data burden 57 19 12 13 100 
Support state program improvement efforts 50 19 15 17 100 
Support local program improvement efforts 56 14 15 4 100 
Data collected are of high quality 38 23 23 17 100 
Advantages outweigh disadvantages 34 28 20 19 100 
Able to collect data on all measures 53 16 14 17 100 
Benefit from state guidance on collecting data 55 24 7 15 100 
Other 6! 32! 1! 60! 100 ! 
IHE 
Impose a substantial data burden 64 14 12 10 100 
Support state program improvement efforts 54 17 15 13 100 
Support local program improvement efforts 58 15 18 10 100 
Data collected are of high quality 38 23 27 13 100 
Advantages outweigh disadvantages 35 25 27 14 100 
Able to collect data on all measures 45 11 30 3 100 
Benefit from state guidance on collecting data 54 20 12 13 100 
Other 12! 26! 0! 62! 100 ! 


Exhibit reads: More than half of local directors agreed that the new measures would impose a substantial data burden. While many 
indicated that they could collect data on all measures (53 percent LEAs, 45 percent IHEs), only 38 percent of local directors said that they 
thought the data collected would be of high quality. 

! Interpret data with caution. These items have a response tate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


States that merged their Tech Prep and basic grants anticipated several obstacles in tracking 
secondary Tech Prep students transitioning into postsecondary education (Exhibit 3.35). 
Many state directors noted the lack of a common student identifier between sectors as a 
challenge (19 secondary, 14 postsecondary). They also reported other issues, such as students 
not self-identifying, lack of ways to share data across sectors, and that the postsecondary sec- 


tor does not, in some states, attempt to identify entering secondary Tech Prep students. 
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Number of states according to obstacles faced in tracking Tech Prep students transitioning 


between secondaty and postsecondary, by education level 


Secondary and postsecondary sectors do not 
share a common student identifier 


Secondaty Tech Prep students do not self- 
identify when matriculating in postsecondary 


Postsecondary institutions do not attempt to 
identify incoming secondary Tech Prep 


students 


Secondary and postsecondary sectors do not 
have a means of sharing data 


Other 


No response 


Secondary 


Postsecondary 


a 
a 
a: 


a 


Oo 
a 
Oo 
NO 
Oo 
Ss) 
Oo 


Number of states 


10 20 30 40 50 


Number of states 


Exhibit reads: Among states that merged their Tech Prep and basic grants, many state directors noted the lack of a common student 
identifier as a challenge (19 secondary, 14 postsecondary). 
NOTE: N = 27 secondary, 27 postsecondary. Exhibit includes responses only from states that reported merging Tech Prep and Basic 


Grant funding in survey question 64. Excludes responses from territories and outlying areas. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


The survey and case studies did not provide further information about the reasons that 
states found the new measures burdensome. The Department did not issue nonregulatory 
guidance for the Tech Prep indicators, which may have posed difficulties for states attempt- 
ing to develop measurement approaches. In an effort to promote comparability and alleviate 
the burden on states, the National Association for Tech Prep Leadership (NATPL) devel- 
oped and shared suggested measurement approaches Tech Prep (National Association for 
Tech Prep Leadership 2009). 


Perkins IV provides specific definitions for Tech Prep secondary and postsecondary students 
(Exhibit 3.36). States reported Tech Prep performance results for the first time in 2008-09. 
At the time of this writing, the Department has not released performance results, and little 


information exists about the utility and quality of Tech Prep indicators and data. 
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Exhibit 3.36. 
Perkins IV: Definitions of Tech Prep students 


Secondary 

Secondary Tech Prep student A secondary education student who has enrolled in two courses in the secondary educa- 
tion component of a tech prep program. 

Postsecondary 

Postsecondary Tech Prep student A student who has completed the secondary education component of a tech prep pro- 


gram; and has enrolled in the postsecondary education component of a tech prep program 
at an institution of higher education. 


Exhibit reads: Perkins IV provides specific definitions for secondary and postsecondary Tech Prep student populations. 
SOURCE: The Carl D. Perkins Career and Technical Education Act of 2006. 


Performance 


Perkins IV requires states to report student performance data each year. States negotiate tar- 
gets for each indicator with the Department, and they update the targets over time to reflect 
data trends and promote continuous improvement. States not reaching at least 90 percent of 
any individual target must develop an improvement plan and may risk losing funds if results 


do not improve over time. 


Por the first time, Perkins IV extends the same level of accountability to local subgrantees. 
LEAs and IHEs now must negotiate performance targets with the state, and if local provid- 
ers fall short of the 90 percent threshold for any measure, they must develop a local im- 
provement plan. State education agencies also now have the authority to withhold Perkins 
funds from LEAs or IHEs that fail to implement an improvement plan, make progtess with- 
in a year of plan implementation, or meet their performance targets for three consecutive 


years (Sec. 123(b)(4)(A). 


Local Perspectives on Performance Requirements 


In the study surveys and case study visits, local subgrantees offered their perspectives on 
changes to performance expectations and Perkins accountability requirements. A majority of 
local subgrantees (61 percent of LEAs and 67 percent of IHEs) reported that introducing 
sanctions would have no impact on local CTE administration and implementation, although 
some (15 percent of LEA and 11 percent of IHE directors) said it would have a somewhat 
to very positive impact. While the surveys did not provide further insight into local sub- 
grantees’ reasons for believing sanctions would have no effect or a positive effect, it is possi- 
ble that the timing of the survey played a part. Local directors responded to the survey in late 
2009, around the time they were submitting the first full year of Perkins IV data to their 
states. They had not yet had much experience with new state policies regarding sanctions for 


failing to meet negotiated targets. 
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About half of local directors (52 percent of LEAs and 54 percent of IHEs) reported that ne- 
gotiating local targets would have no impact on CTE administration and implementation, 
and about one-quarter thought it would have a somewhat to very positive impact (Exhibit 
3.37). This is consistent with the information presented later in this chapter (Exhibit 3.43 
and Exhibit 3.44), which suggests that, among LEAs and IHEs that negotiated performance 
targets with their states, about half to about one-third of LEA and IHE directors experi- 


enced little to no difficulty in the negotiation process for each indicator. 


Exhibit 3.37. 


Percentage of LEAs and IHEs that reported that they expected various changes to 


Perkins IV accountability requirements to have a positive or negative impact on local 


LEA 


THE 


administration and implementation 


@ Somewhat to very positive ™Noimpact Somewhat to very negative 


Requitement to report disaggregated outcomes 


for populations in ESEA oe ee 
Sanctions for failing il) al 
Requirement to negotiate targets with state aT 54 


Requitement to report disaggregated outcomes 
for specific populations 


Sanctions for failing 11 67 


ie) 
on 
a 
i) 


oO 
NO 
oO 
as 
oO 
aD 
oO 
[ee] 
oO 
a= 
S 
oO 


Percent 


Exhibit reads: About half of LEAs and IHEs reported that negotiating local targets and reporting on disaggregated populations would 
have no impact on local administration and implementation. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


About three-quarters of LEAs and IHEs indicated that reporting on disaggregated popula- 
tions would have no impact or a positive impact on how they administer local CTE pro- 
grams, while about one-quarter said it would have a somewhat to very negative impact. This 
may reflect the fact that reporting outcomes for special populations was a requirement of 
Perkins III, and the only change to the special population definitions for Perkins IV was the 
removal of the Perkins III reference to “individuals with other barriers to educational 
achievement” (Perkins III Sec. 3(23) and Perkins IV Sec. (3(29)). Perkins IV adds, however, 
the requirement that each eligible local recipient must report disaggregated secondary indica- 
tor data annually using the categories of students listed in ESEA (Sec. 1111(h)(1)(C)@). 
Most of the ESEA categories are identical to the Perkins IV special populations, but ESEA 
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also includes “migrant status.” While only 4 percent of LEA directors had no confidence in 
the accuracy of the data they reported for migrant students (72 percent had at least some 


confidence), 23 percent did not know if they had confidence in these data (Exhibit 3.16). 


Case study site visits offered an opportunity to discuss Perkins IV accountability require- 
ments in more depth with state and local administrators. One local postsecondary adminis- 
trator raised issues about whether responsibility for defining accountability requirements 
rests with the state agency or the federal government: “Seems that the eligible agency is get- 
ting instructions from [the Department], and they interpret what they’re hearing. Then they 
give it to us, and then we’re trying to interpret what we’re hearing, and it can be confusing.” 
This administrator also mentioned the impact of even small changes to the system: “We do 
what we think we’re supposed to do, and then the wording gets changed slightly and we 


have to go back and redo it all. It truly is a nightmare.” 


Among case study states, state and local administrators said that the cost of meeting the Per- 
kins IV accountability requirements is prohibitive and that they do not have either the staff 
or funds needed to meet all the requirements. More than one suggested that the accountabil- 
ity requirements ate an unfunded mandate. A state director and two local directors noted 
that they had participated in discussions about possibly giving up Perkins funds because of 


the burden imposed and the complexity of reporting requirements. 


One superintendent had concerns about the demands on teachers’ time: “It’s the teachers in 
the classroom [who] are doing all of this .. . and it takes away directly from .. . what they 
can do with students. I know we live in a time of accountability, but the paperwork is a kill- 
er—and that’s on everything, not just Perkins.” While another state administrator observed 
that the state budget crisis and staff cuts make data collection and reporting a challenge, he 
said that it was unrealistic for the state to collect millions of dollars in federal funds “and not 


have to be accountable for [them].” 


Although these case study responses may seem to contradict the survey responses, there are 
at least two possible explanations for the disparity. First, the case study states may have had a 
disproportionate and nonrepresentative number of individuals who believed the changes to 
Perkins IV would have a negative impact or increase the burden on states and local subgrant- 
ees. Second, the survey did not ask whether the Perkins IV accountability framework posed a 
burden, only what impact the changes would have. It is possible that states may see little addi- 
tional positive or negative impact from the changes but still feel that the overall accountabil- 


ity requirements ate burdensome. 
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Setting Performance Targets 


For the 2007-08 transition year, the Department and states negotiated performance targets 
only for the secondary academic attainment and graduation rate measures (Justesen 2007a). 
States negotiated targets for the remaining measures as part of their five-year plans and can 


negotiate new targets prior to the third and fifth program years. 


Perkins IV requires the Department to consider the following two key issues when negotiat- 


ing performance levels with states (Sec. 113(b)(3(A) (vi): 


e How the proposed targets compare to targets for other states, taking into ac- 


count participant and program characteristics; and 


e The extent to which the proposed targets will encourage continuous improve- 


ment on each indicator. 


State negotiations with the Department 
State and local directors reported generally positive experiences in negotiating performance 
targets, noting that similar issues were important to them during the process. Postsecondary 
state directors and IHE directors appeared to have had more concerns and challenges than 


their secondary counterparts. 


State directors said that past Perkins performance and state goals for program performance 
were the most important issues considered when negotiating targets with the Department 
(Exhibit 3.38). Many considered multiple issues during the process, including anticipated 
changes in state circumstances, changes in funding, program modifications, and changes in 


student populations. 


°5'The Department allows states to request performance-level revisions in other years “if the State can 
show that an unanticipated circumstance arose in the State that resulted in a significant change in the 
factors that were considered” when the state originally negotiated targets (Dann-Messier 2009, p. 2). 
“Unanticipated circumstances may include: significant shifts in population, economic changes such as 
spiraling unemployment rates, or natural disasters that closed programs for significant periods of 


” 


time. 
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Exhibit 3.38. 
Number and percentage of states according to the importance of various factors when 


negotiating performance targets with the Department, by education level 


(5—point scale from Not important to Very important) 


Number Percent 
Mote than 
not, less 
Very than very Not At least 
important important important Don’t No some Not 
Factors (5) (2 - 4) (d) know response Total | importance important Total 
Secondary 
Past state 
performance 37 12 0 1 1 51 100 0 100 
State program 
performance goals 34 14 0 2 1 51 100 0 100 
Anticipated changes 
in the state, excluding 
funding 21 21 7 1 1 51 86 14 100 
Anticipated changes in 
CTE programs 12 27 9 2 1 51 81 19 100 
Anticipated changes in 
student populations 8 32 8 2 1 51 83 17 100 
Anticipated change in 
state or local funding 16 25 7 2 1 51 85 15 100 
Other 3 0 1 3 44 51 75 25 100 
Postsecondary 
Past state 
performance 36 8 1 1 2 48 98 2 100 
State program 
performance goals 35 8 1 2 2 48 98 2 100 
Anticipated changes 
in the state, excluding 
funding 18 22 7 0 1 48 85 15 100 
Anticipated changes 
in CTE programs 6 27 12 2 1 48 73 27 100 
Anticipated changes 
in student populations 9 23 12 3 1 48 73 27 100 
Anticipated change in 
state or local funding 10 24 12 0 2 48 74 26 100 
Other 4 1 1 2 40 48 83 17 100 


Exhibit reads: Past state Perkins performance (37 secondary, 36 postsecondary) and state goals for program performance (34 secondary, 
35 postsecondary) were very important issues when negotiating targets with the Department. 

NOTE: Percentage calculations exclude “Don’t know” and “No response.” 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Mote than half of secondary state directors had little or no difficulty in negotiating their 
transition year and annual performance targets with the Department (Exhibit 3.39). Among 
postsecondary directors who answered the survey, however, 23 indicated that it was some- 
what or very difficult to negotiate annual targets. States noted a lack of baseline data as a 
common difficulty in setting performance benchmarks and targets for both the secondary 
(17) and postsecondary (17) sectors (Exhibit 3.40). 


Exhibit 3.39. 
Number of states according to the level of difficulty encountered when negotiating state 
performance targets with the Department, by education level 


(5-point scale from Not difficult to Very difficult) 


m Little or no difficulty (1-2) | Somewhat to very difficult (3-5) Don’t know/No tesponse 


Number of states 


50 
40 4 
4 29 
] 24 93 
20 4 
10 4 
? 1 
0 4 
Transition year targets Annual targets Transition year targets Annual targets 
Secondary Postsecondary 


Exhibit reads: More than half of secondary state ditectors reported little or no difficulty in negotiating their transition year and annual 
performance targets with the Department. Among postsecondary state ditectors, 23 indicated that it was somewhat or very difficult to 
negotiate annual targets. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Exhibit 3.40. 


Number of states according to various reasons for encountering difficulty when negotiating 


155 


state performance benchmarks and targets with the Department, by education level 


Insufficient baseline data 


Repercussions of failing to meet targets 


Anticipated difficulty gatheting/ 


Secondary 


ae 
a 
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Postsecondary 
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submitting data 


Anticipated difficulty negotiating 
targets with LEAs/IHEs 
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No response | 0 0 


Lack of clear guidance from Department 
on continuous improvement 


Other 


Number of states Number of states 


Exhibit reads: Seventeen secondary and 17 postsecondary state directors noted a lack of baseline data as a common difficulty in setting 


performance benchmarks and targets. 
NOTE: N = 27 secondary, 27 postsecondary. Exhibit excludes states that did not report encountering difficulty. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


The case study site visits revealed differing views on the experience of negotiating perfor- 
mance levels with the Department. While some state administrators praised the Depart- 
ment’s efforts, others described challenges. One state administrator commended the 
negotiation process as “very professional,” explaining, “We made some suggestions, ex- 
plained why we made those suggestions, and [the Department] responded.” A postsecondary 
administrator offered another positive view, “We didn’t get our way the first time, but we 
talked about what’s reasonable. What could we look at as far as a growth model? ’'m very 
pleased; working with [Department staff person] has been good.” 


One state administrator was frustrated by the lack of clear direction from the Department on 
continuous improvement: “It’s not really clear what methodology they’re [using to select] a 
target. If it was more transparent [and] ... we knew what their goal was, we . . . could figure 
out what to do....” This administrator questioned the value of trying to achieve perfect 
outcomes (100 percent) on performance measures: “The negotiations . .. were based on 


reaching 100 percent. ... We all know that was not a reasonable supposition that anyone will 


CHAPTER 3. ACCOUNTABILITY 156 


ever reach 100 percent in [the] eight-year period that was negotiated under [ESEA], but we 


were held to those [targets].” 


A state assessment administrator noted difficulty with the process, describing it as “pro- 
posal-counter proposal conducted via email with the regional accountability specialist [RAS]. 
... There was apparently no room for negotiation on 181, 182, and 4S1 with our RAS. It 
was quite clear that we were going to be held accountable for the [ESEA] standards that had 
not been negotiated for CTE ... and we did capitulate.” 


Another state director described an attempt to “fund a statistical review to map out a five- 
year plan of what would be realistic for our cohort of CTE students, but [the Department] 
wasn’t interested in that approach.” The state and the Department eventually came to a 


compromise, but only after several rounds of negotiation. According to the administrator: 


[Our] population is not your typical secondary school population; we’re 
oversubscribed in special education. So [our RAS] got that [we] couldn’t 
just follow [ESEA] growth charts, yet still negotiated fairly. For instance, 
for math in the CTE population, we were at 11 percent proficiency. ... We 
had negotiated 20 or 22 percent proficiency, even though we knew we were 
only at 11 percent, then they came back and wanted 35 or 37 percent. ... 
Neither our agency nor [the Department] would budge. I said, “Then I 
don’t know what options we have here, except we’re not going to take the 
money if it’s based on this. This is crazy. I can’t agree to this.” ... It was 


not a rational discussion at all. 


Another state director encountered resistance from the Department in negotiating perfor- 

mance levels based on what the state perceived to be realistic. The director explained, “We 
want to be challenged by the [performance levels], but don’t make them unrealistic. In our 
negotiation, I think the comment was made, “This is what is expected, so I really can’t even 


negotiate with you’.” 


State negotiations with LEAs and IHEs 
States adopted a variety of approaches to setting local targets with LEAs and IHEs sub- 
grantees. Some required all local providers to meet the state-negotiated targets, while others 
negotiated targets with individual subgrantees. In some states, all local providers adopted the 
state targets (18 secondary, 19 postsecondary), while in other states at least one local provid- 


et negotiated a different target for one or more measures (22 secondary, 16 postsecondary) 
(Exhibit 3.41). 
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Exhibit 3.41. 
Number of states according to whether they negotiated different targets on at least one 


measute with one or more local subgrantees for 2008-09, by education level 


Number of states 
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Alladopted — At least Don’t No Alladopted = At least Don’t No 
state- one LEA know response state- one IHE know response 
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levels different levels different 
performance performance 
level on at level on at 
least one least one 
measure measure 
Secondary Postsecondary 


Exhibit reads: In some states, all local providers adopted the state targets (18 secondary, 19 postsecondary), while in other states at least 
one local provider negotiated a different target for one or more measures (22 secondary, 16 postsecondary). 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Three case study states did not negotiate targets with local subgrantees. In one state, an ad- 
ministrator said: “We have defaulted to not really negotiating, but it’s more requiring that 
they meet a state goal.” One postsecondary administrator described the state’s approach as 
“dictated negotiation.” Local providers can choose to adopt the state performance level for a 


measure, or they may add 3 percent to their current performance level for each measure. 


Another state staff person said, “Negotiating with locals is ridiculous. Frankly, we didn’t do 
it. We told our locals you have to accept the state levels or ask to negotiate because that’s 
what the law says.” The state based its approach on problems it perceived with local negotia- 
tions: (1) lack of state staff time to conduct negotiations with local providers and (2) compli- 
cations that would arise if the state lowered targets for some providers and not others. The 
administrator went on to say, “For every agency I negotiate a lower number, that puts the 
state target in jeopardy. If you truly looked at the core indicators of quality programs, why 
would I hold somebody else to a different standard?” 


CHAPTER 3. ACCOUNTABILITY 158 


In one case study state that does allow negotiation of different targets, the state and local 

J 8 gets, 
providers have collaborated to develop an “online system that locals can access to see their 
performance measures. To accept, they just click. To negotiate, they... submit an official 


request to negotiate, with supporting justification on that particular measure.” 


State directors cited program performance goals and past performance as important issues 
when negotiating performance targets with LEAs and IHBs, issues similar to those consid- 


ered important during state negotiations with the Department (Exhibit 3.42). 


CHAPTER 3. ACCOUNTABILITY 159 


Exhibit 3.42. 
Number and percentage of states according to the importance of different factors when 
negotiating performance targets with local subgrantees, by education level 


(5—point scale from Not important to Very important) 


Number Percent 
More than 
not, less than 
Very vety Not At least 
important important (2- important Don’t No some Not 
Factor (5) 4) (4) know response Total | importance important Total 
Secondary 
Program performance 
goals 15 6 0 1 0 22 100 0 100 
Past performance 19 3 0 0 0 22 100 0 100 
Anticipated changes 
in local conditions 7 12 3 0 0 22 86 14 100 
Anticipated changes 
in CTE programs 3 14 5 0 0 22 77 23 100 
Anticipated changes in 
student populations 3 13 6 0 0 22 73 27 100 
Anticipated changes in 
state or local funding 4 10 8 0 0 22 64 36 100 
Other 1 0 0 1 20 22 100 0 100 
Postsecondary 
Program performance 
goals 9 7 0 0 0 16 100 0 100 
Past performance 9 7 0 0 0 16 100 0 100 
Anticipated changes in 
local conditions 6 9 0 0 1 16 100 0 100 
Anticipated changes in 
CTE programs 1 12 2 1 0 6 87 13 100 
Anticipated changes in 
student populations 2 10 3 1 0 16 80 20 100 
Anticipated changes in 
state or local funding 4 8 3 0 1 16 80 20 100 
Other 3 1 0 0 12 16 100 0 100 


Exhibit reads: State directors cited program performance goals (21 secondary, 16 postsecondary) and past performance (22 secondary, 16 
postsecondary) as important issues when negotiating performance targets with LEAs and IHEs. 

NOTE: Exhibit includes only those states that negotiated at least one different performance target with at least one LEA or JHE. 
Percentage calculations exclude “Don’t know” and “No response.” 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Secondary local directors had little or no difficulty in negotiating targets on most individual 
measures with their states, with the exception of the nontraditional measures (Exhibit 3.43). 
About one-quarter of LEA directors indicated that it was at least somewhat difficult to reach 
agreement on the nontraditional participation and completion measures. At least 20 percent 
of IHE directors reported it was at least somewhat difficult to negotiate targets for every 
measure except technical skill attainment (Exhibit 3.44). Postsecondary local directors said 
that negotiating targets for nontraditional participation and completion posed the greatest 
difficulties. A large percentage of local directors did not know how to respond to this ques- 
tion, which may indicate a lack of familiarity with the process or not having the option to 


negotiate different performance levels. 


Exhibit 3.43. 
Percentage of LEAs according to the level of difficulty experienced when negotiating 
performance targets with the state 


(5-point scale from Not difficult to Very difficult) 


@ Little or no difficulty @ Somewhat to very difficult @ Don’t know 
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Exhibit reads: About one-quarter of LEA directors indicated that it was at least somewhat difficult to reach agreement on the 
nontraditional participation and completion measures. 

NOTE: Estimates were weighted to reflect population values. Detail may not sum to totals because of rounding. 

SOURCE: LEA Survey, 2009. 
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Exhibit 3.44. 
Percentage of IHEs according to the level of difficulty experienced when negotiating 
performance targets with the state 


(5-point scale from Not difficult to Very difficult) 


w Little or no difficulty & Somewhat to very difficult @ Don’t know 


Technical skill attainment (1P1 
Completion (2P1 
Retention or transfer (3P1 


Placement (4P1 
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0 20 40 60 80 100 


Percent 


Exhibit reads: IHE directors said that negotiating targets for nontraditional participation and completion posed the greatest difficulties (27 
petcent each). 

NOTE: Estimates were weighted to reflect population values. Detail may not sum to totals because of rounding. 

SOURCE: IHE Survey, 2009. 


Postsecondary local directors indicated that gathering and submitting the data required for 
Perkins accountability was their most common concern when negotiating performance tat- 
gets with their states (Exhibit 3.45). Local directors from both education levels expected to 
face challenges in meeting the targets (61 and 55 percent, respectively), and IHE directors 
expressed more concern about insufficient baseline data (50 percent) than did LEA directors 


(34 percent). 
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Exhibit 3.45. 
Percentage of LEAs and IHEs according to various reasons for encountering difficulty 


when negotiating performance targets with state 


LEA IHE 


Anticipated difficulty meeting targets | 61 | 55 
Anticipated difficulty gathering/ 
submitting data 


Lack of state guidance on continuous 
improvement 


Insufficient baseline data ae 34 | 50 
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Exhibit reads: When negotiating performance targets with the state, LEAs and IHEs anticipated difficulty in meeting the targets (61 and 
55 percent, respectively) and gathering and submitting data (49 and 62 percent, respectively). 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


Local providers interviewed during the case study visits reported various experiences when 
negotiating performance targets with their states. One administrator was concerned about 
delayed funding: “They came to us and said ‘Here are your performance measures. Here’s 
what we’re looking at.’ You either accepted the percentages that they said you’re going to 
meet, or you negotiated with the state. If you negotiated with the state, it delayed the Perkins 
funding for your district.” In the same state, a postsecondary administrator noted, “There 
were only two colleges in the state that negotiated. ... We got the pretty clear message that 


there wasn’t much room [for negotiation].” 


In contrast, a secondary respondent in one state claimed, “The negotiations are good be- 
cause [they] cause some of the old directors to be more accountable . . . and make them bet- 
ter understand the process. When they have to sit down at the table and they have to be able 
to talk intelligently about it... . I think we’re in a time where that is going to be... the big 
white horse in the room that everybody’s going to have to deal with. . . . I like the way the 


state goes about it: let’s negotiate so you can meet it.” 
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More than one local respondent expressed a desire for a more transparent process of calcu- 
lating targets. According to one administrator, it would be beneficial to know “not just what 


the formula is, but how the state got the numbers for each institution.” 


Reaching Performance Targets 


Perkins IV requires states that do not meet at least 90 percent of their target on any individu- 
al measure to develop an improvement plan. If the state fails to implement a plan, to show 
improvement within a year of plan implementation, or to meet at least 90 percent of the ne- 
gotiated target on a single indicator for three consecutive years, the Department may with- 
hold all or part of the state’s Perkins funds. Perkins IV requires local subgrantees to 


implement a local improvement plan if they do not meet 90 percent of any target. 


The timing allowed for developing and implementing an improvement plan may pose a chal- 
lenge for both states and local providers, however. An administrator in one case study state 
explained, “It doesn’t make sense to have subgrantees put together plans and have only six 
months to influence data that get re-evaluated the next year.” This state’s solution was to ex- 
tend the scope of improvement plans to two years for local providers, and notify them that 


they “have to be working toward the targets in the first year.” 


Most states have policies to sanction local providers who do not meet Perkins IV perfor- 
mance levels (Exhibit 3.46). Nearly all states (47 secondary, 38 postsecondary) required 
LEAs and IHEs to develop a local improvement plan in that circumstance, while others (30 
secondary, 22 postsecondary) reserved the option of restricting the flexibility of Perkins funds 
use. Very few states withheld or decreased state and Perkins funds for local providers, how- 
ever (11 secondary, 9 postsecondary). In addition, very few had no consequences for local 


providers who fall short of negotiated targets. 
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Exhibit 3.46. 
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Number of states according to consequences for local subgrantees failing to meet one or 


mote performance targets, by education level 
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Exhibit reads: Most secondary (47) and postsecondary (38) state directors reported that they would require LEAs and IHEs that failed to 
meet a performance target to develop a local improvement plan, while 30 secondary and 22 postsecondary state directors said they reserved 
the option of restricting the flexibility of the use of Perkins funds. 


NOTE: N = 51 secondary, 48 postsecondary. 


SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Representatives from two case study states cited data quality as a barrier to implementing ef- 


fective sanctions. According to one secondary administrator, “Occasionally ... we have a data 


reporting problem that has to get fixed that may have affected performance.” In the second 


state, a postsecondary administrator described a situation in which an error in the state data 


system was found, resulting in the suspension of sanctions implemented the previous year. 


More than one state expressed reservations about using sanctions for local providers. One 


respondent said, “When it was about program improvement and not sanctions, give me a 


carrot and I can lead them on. Give me sanctions, and they just don’t want to participate an- 


ymore, which means that I have less influence over what they’re doing in their programs be- 


cause they don’t want my money.” This state seemed to be working proactively to avoid 


using sanctions: “What we want to do is provide... the help and assistance you need, so 
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you're not sanctioned. That’s why we set up this data monitoring process and are helping the 


locals clear up their data.” 


Similarly, another administrator observed, “In the grand scheme of things, [Perkzns funding] 
is not very much money for a number of the districts. For them to go through the process of 
submitting data, following through with all of the other assurances and requirements associ- 
ated with it, and then be flogged publicly and told you can’t use your money for this particu- 


lar purpose without fixing something, it undermines the process.” 


Another state had not yet developed a formal process or policy regarding sanctions, although 
the state does limit funding flexibility for underperforming local subgrantees. A respondent 
in this state offered her perspective on working with LEAs and IHEs to improve perfor- 
mance and prevent sanctions: “What we said is that, rather than taking money away, which 
doesn’t make sense if you’re struggling, is that we would have them focus their funds. ... 
They wouldn’t lose those Perkins funds, but they would be obligated to spend either all... or 


a portion .. . on improving the performance data that are falling behind.” 


According to case study interviews, two of the six states have created a system of financial 
sanctions for underperforming programs. One state director explained that any local provid- 
ers who do not reach the required performance threshold for two years in a row “will no 


longer be included as CTE programs eligible for continued Perkins funding.” 


Some states are actively pursuing a system of rewards for high performance. According to 
one state administrator, “We want to move down the road to more of a performance-based 
carrot approach, in that you’re rewarded for doing well, as opposed to smacking people over 
the head or somehow publicly flogging people for not doing well.” Two state administrators 
indicated interest in adopting performance-based funding or reward systems, with one ex- 
plaining, “Our ultimate goal is to reach a point where we can provide performance-based 


funding [for] school districts and/or community colleges.” 


According to local directors responding to the study surveys, program improvement plans 

were the most likely sanction their states might use in the event of missing a negotiated tar- 
get (Exhibit 3.47). Local directors reported that professional development was also a com- 

mon approach among states. Very few said that their states would eliminate 


underperforming programs (8 percent of LEAs and 12 percent of IHEs). 
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Exhibit 3.47. 
Percentage of LEAs and IHEs that reported the presence of state policy on the use of 


various corrective actions for addressing accountability deficiencies 
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Exhibit reads: LEAs (65 percent) and IHEs (78 percent) said that program improvement plans were the most likely sanction their states 
might use if they missed a negotiated target. 

! Interpret data with caution. These items have a response rate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


Local administrators in some case study communities were concerned about the possibility 
of financial consequences. One explained: “If we don’t write the plans, we won’t get next 
yeat’s Perkins money. So there is a money sanction tied to it.” Some administrators of LEAs 
and IHEs took a more favorable view of potential financial sanctions. Although he did not 
want to lose funding, one administrator said, “If you don’t meet one, you get time to submit 
an improvement plan. .. . It’s strict, but reasonable.” Another secondary administrator ap- 
peared to discount financial sanctions as an insurmountable barrier: “I have heard of sanc- 


tions, but if we lose $150,000 we will continue to do good things.” 
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Using Data 


The ultimate purpose of the accountability system is to promote the best possible education- 
al outcomes for students. By refining, expanding, and clarifying the accountability frame- 
work, Congress and the Department are supporting state and local efforts to collect better 
data and to report and use state and local performance results to improve CTE programs 


and student outcomes. 


Reporting 


The Department shares Perkins accountability results through an annual Report to Congress on 
State Performance (U.S. Department of Education 2010b). In addition to detailing state pro- 
gress in meeting performance targets, the report provides information about the Depart- 


ment’s efforts to promote valid and reliable data collection and reporting. 


States also share information, disseminating performance results to local providers through a 
variety of methods (Exhibit 3.48). As of 2008-09, most states released statewide perfor- 
mance outcomes averaged across all secondary or postsecondary subegrantees (40 secondary, 
29 postsecondary). While states publicized Perkins performance data at the local level, very 
few (9 secondary, 6 postsecondary) controlled for similarities in institutional characteristics. 
Fewer than half of states posted performance reports on the Internet. Three secondary and 


six postsecondary state directors reported that their states did not share performance data at 
all for LEAs and IHEs. 


One state administrator described significant progress in communicating performance tre- 
sults: “Until 07-08, [our state] only reported aggregate data at the state level; that was the 
first year the state began reporting school-level data. One of the big projects of our state 
longitudinal data system grant is a decision-support system, where you're able to mine this 
big monster data queue and go in and look at data and slice and dice it however you would 
like... .” In another state, postsecondary administrators noted that they have made substan- 
tial changes in the way the state shares information with IHEs. In the past, the state reported 
only ageregate data and did not share information from individual providers. Now the state 
provides performance reports to individual institutions, so providers can compare their re- 


sults to the aggregate statewide performance results. 
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Exhibit 3.48. 
Number of states according to the use of various methods to share performance outcomes 


with local subgrantees, by education level 
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Exhibit reads: Most state directors reported that as of 2008-09, their states released statewide performance outcomes, averaged across all 
LEAs ot IHEs (40 secondary, 29 postsecondary). 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Several administrators noted the various ways that data are used at local and state levels. In 
one case study state, a respondent noted that analyzing performance results enables staff to 
map out deficiencies and work to boost enrollment and scores. Another administrator indi- 
cated an interest in identifying high schools sending underprepared students to area CTE 


centers. 


More than half of LEA and IHE directors reported receiving institution-level performance 
data (54 percent and 61 percent, respectively) (Exhibit 3.49). Approximately two-thirds said 
they received information containing statewide averages for all LEAs and IHEs. A small 
percentage of LEA (15 percent) and IHE (10 percent) directors reported that their state 


provided no feedback on institutional performance. 
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Exhibit 3.49. 
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Percentage of LEA and IHE directors that reported receiving various types of institutional 


performance data from the state 
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Exhibit reads: More than half of local directors reported receiving institution-level performance data (54 percent of LEAs and 61 percent 


of IHEs). Approximately two-thirds of local providers said they received information containing statewide averages for all local 


subgrantees. 


NOTE: Estimates were weighted to reflect population values. 


SOURCE: LEA and IHE Surveys, 2009. 


Program Improvement 


Most states used annual Perkins results to identify programs needing improvement and to 


provide targeted technical assistance (Exhibit 3.50). Although few states (11 secondary, 4 


postsecondary) used core indicator data to reward LEAs and IHEs that performed well, 


more than half (29 secondary, 28 postsecondary) provided additional program improvement 
resources based on Perkins indicator results. Twenty-one states used the data at least some- 
what to sanction LEAs for low performance. Sanctions were much less popular for postsec- 
ondary, with only 12 states employing sanctions at least somewhat against low-performing 
THEs. 


It is notable that while most state directors reported using Perkins data to provide targeted 
technical assistance (45 secondary, 40 postsecondary), only about one-third of local sub- 
grantees thought their state had a policy of providing professional development to staff in 
underperforming programs (Exhibit 3.47). 
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Exhibit 3.50. 
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Number of states according to the extent to which they used Perkins core indicator data, by 


education level 


(5—point scale from Used to a great extent to Used not at all) 


Used to a great 
extent to used Used not much 


somewhat to not at all 

Use of data (1 - 3) (4-5) No response Total 

Secondary 
Identify programs needing improvement 47 3 1 51 
Provide targeted technical assistance 45 5 1 51 
Identify underserved special populations 36 14 1 51 
Identify unusually effective programs 35 15 fl 51 
Provide resources for program improvement 29 21 1 51 
Sanction low performing LEAs 21 29 1 51 
Reward LEAs for high performance a 38 2 51 
Other 3 3 45 51 

Postsecondary 
Identify programs needing improvement 41 6 1 48 
Provide targeted technical assistance 40 7 il 48 
Identify underserved special populations 34 13 1 48 
Identify unusually effective programs 29 18 1 48 
Provide resources for program improvement 28 19 1 48 
Sanction low performing IHEs 12 34 2 48 
Reward IHEs for high performance 4 43 1 48 
Other 3 6 39 48 


Exhibit reads: Most states used annual Perkins results to identify programs needing improvement (47 secondary, 41 postsecondary) and 


to provide targeted technical assistance (45 secondary, 40 postsecondary). 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


One case study state reported using instructor-level data to improve teacher performance: 
“We have been able to look at programs and see if we thought the teachers were doing a 
good job and make recommendations for professional development in a much more pre- 
scriptive way than we’ve been able to in many years.” This state also is developing a report- 
ing mechanism to produce customized performance reports: “One of the things that we 
hope to launch for Perkins is a business improvement project. ... We have the [reports] by 
school, by program, and then the sub-indicators that we hope to be able to share to give 


[grantees] the resources they need to identify problems.” 


Among states that do not currently use performance data to identify underperforming pro- 


grams, several will use the data for this purpose in the next two years (one secondary, five 
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postsecondary) (Exhibit 3.51). A few planned to use Perkins performance data to reward 


LEAs (four states) and IHEs (four states) in the next two years. Among those not currently 


using sanctions, eight secondary and eight postsecondary planned to implement sanctions 


for low-performing LEAs and IHEs within the next two years. 


Exhibit 3.51. 
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Number of states that are not already using Perkins data in various ways that may use 


Perkins data in those ways in the next two years, by education level 


Don’t No Already 
Use of data Yes No know response using data 
Secondary 
Sanction LEA for low performance 8 4 9 1 29 
Reward LEA for high performance 4 11 16 0 20 
Provide resources for program improvement 3 3 6 0 39 
Identify unusually effective programs 3 0 4 0 44 
Identify programs needing improvement 1 0 1 0 49 
Provide targeted technical assistance 0 0 2 0 49 
Identify underserved special populations 0 1 2 0 48 
Other 0 1 0 1 49 
Postsecondary 
Sanction IHE for low performance 8 8 11 0 21 
Identify programs needing improvement 5 0 0 0 43 
Provide resources for program improvement 5 4 0 36 
Reward THE for high performance 4 17 14 0 13 
Identify unusually effective programs 4 1 3 0 40 
Provide targeted technical assistance 3 1 0 0 44 
Identify underserved special populations 3 2 1 0 42 
Other 1 0 2 3 42 


Total 


O Or OW OL oO OI OM 


48 
48 
48 
48 
48 
48 
48 
48 


Exhibit reads: Among states not currently using sanctions, eight secondary and eight postsecondary directors said they plan to implement 


sanctions for low-performing LEAs and IHEs within the next two years. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Using Perkins IV performance data to identify underperforming programs was common 


practice in the case study states. Five state representatives described their approaches to 


spotting local providers who did not meet at least 90 percent of any target, and one outlined 


a color-coded system for categorizing local providers who met targets, were within 90 per- 


cent of the target, or fell below the target. 


One state uses a “risk-based monitoring” system, in which it more closely monitors local 


providers falling below 90 percent of the performance target. The state required the provid- 


ers to prepare corrective action plans focused on the problematic indicators. A postsecond- 
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ary administrator outlined plans to promote local subgrantee review and use of data. While 
some providers meet the negotiated targets for indicators, they may not meet targets for dis- 
ageregated subgroups, particularly for the nontraditional indicators. According to an admin- 
istrator, “We even make them count how many of these special population groups are below 
their district-negotiated targets in their programs. We could do it electronically, but we make 


them count just so we know they looked.” 


About half of LEAs and IHEs used their data quite a bit or to a great extent to identify pro- 
grams needing improvement and make funding choices in 2008-09 (Exhibit 3.52 and Exhib- 
it 3.53). About 40 percent of LEAs and IHEs used the data to offer targeted technical 
assistance and about one-third used it to identify underserved populations. Among LEAs 
and IHEs that did not use a given approach, approximately one-quarter to one-third planned 


to employ an approach in the future.*° 


Exhibit 3.52. 
Percentage of LEAs according to the extent to which they used prior year Perkins data for 
various purposes in 2008-09 


(5-point scale from Used not at all to Used to a great extent) 


@ Used quite a bit to a great extent Used not much to somewhat Used not at all 


Identify programs needing improvement 


Make program funding decisions 


Provide targeted technical assistance 


Identify underserved special populations 


Other 


0 20 40 60 80 100 


Percent 


Exhibit reads: About half of LEAs used Perkins data quite a bit or to a great extent to identify programs needing improvement and make 
program funding decisions (49 percent each) in 2008-09. 

! Interpret data with caution. These items have a response rate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA Survey, 2009. 


°6 LEA and IHE Surveys, 2009. 
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Exhibit 3.53. 
Percentage of IHEs according to the extent to which they used prior year Perkins data for 
various purposes in 2008-09 


(5-point scale from Used not at all to Used to a great extent) 


m Used quite a bit to a great extent Used not much to somewhat Used not at all 


Identify programs needing improvement 


Make program funding decisions 


Provide targeted technical assistance 


Identify underserved special populations 


Other 


0 20 40 60 80 100 


Percent 


Exhibit reads: Many IHEs used Perkins data quite a bit or to a great extent to identify programs needing improvement (47 percent) and 
make program funding decisions (48 percent) in 2008-09. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: IHE Survey, 2009. 


Although survey results indicated that local subgrantees use Perkins data for purposes like 
identifying programs that need improvement, conversations in case study states revealed that 
at least a few providers do not use these data at all. One person said, “The feds told us we 
have to report something, so here are the numbers. . . . If the goal is really to improve our 
programs, I’m all for that, but I need accurate information in order to do that.” A second local 
provider viewed Perkins accountability data as a compliance requirement: “I do what’s asked 
for, get [the data], look at them, and report them rather than act on them.” A third said that, 
because the district is performing well, there was no need to use the performance data to im- 
prove. This administrator explained: “If it should come to pass in the next application that we 
have to address a certain area, then .. . I will photocopy [the data] and see if any of my teach- 
ers want to look at them.” Several postsecondary providers noted that Perkins data were a 


starting point and that they need to perform other analyses to report more useful local data. 


A local secondary administrator, however, said that both administrators and instructors in 
the district use Perkins data to learn about program performance. An IHE representative 
said, “We use all of those data in terms of our program improvement. We also use those da- 
ta to close programs... . I would also say we use them to direct resources.” In one state, a 


local respondent reported using Perkins data extensively: “We brought all the principals to- 
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gether and shared the data with them so they could see the progress that their CTE students 
have achieved.” In the same state, another local director said, “In our types of programs... 
we get kids who are right on the cusp... . It’s the data that help us know how to help those 

kids and push those kids in the direction they need to go.” 
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Chapter 4. Programs of Study 


The Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV) introduced the tre- 
quirement that all local subgrantees offer programs of study (POS) that incorporate academ- 
ic, career, and technical content to prepare students for the transition from secondary to 
postsecondary education and careers. While the POS requirement is new, the idea of organ- 
izing instructional delivery to make the knowledge and skills studied in school more relevant 
to students’ career goals is a time-tested concept. In the last quarter of the 20th century, ini- 
tiatives such as school-to-work, apprenticeships, dual enrollment, and Tech Prep were intro- 
duced to strengthen the links between secondary and postsecondary education and between 


education and work (Lewis, Kosine, and Overman 2008). 


Perkins IV builds upon these earlier initiatives, and on statutory language contained within 
Title HW (Tech Prep) of the 2006 legislation, in putting forth POS as an organizing principle 
for delivering CTE course work.! Although all local subgrantees are required to offer at least 
one POS, Perkins IV offers relatively little direction on how states are to support local educa- 
tion agencies (LEAs) and institutions of higher education (IHEs) in developing and imple- 
menting these programs. And because the U.S. Department of Education (the Department) 
did not issue policy guidance on POS until January 2010, states have been left largely on 
their own to interpret statutory language and provide policy support, guidance, and technical 
assistance to local subgrantees. Accordingly, the analyses in this chapter seek to answer the 


following research questions: 


e How are states and local subgrantees developing and implementing POS? 


e What challenges are states and local subgrantees facing in developing and im- 


plementing POS? 


e What is the scope of student participation in POS? 


! Tech Prep was introduced as a demonstration program in the Car/ D. Perkins Vocational and Applied 
Technology Education Act of 1990 (Perkins I] and given increased emphasis in the Car/ D. Perkins Voca- 
tional and Applied Technology Education Act of 1998 (Perkins Il). Appropriations for the program were 
discontinued in FY 2011.’Tech Prep programs combine at least two years of secondary and two years 
of postsecondary education with course work offered in a nonduplicative, sequential manner. Aca- 
demic instruction and technical skill instruction are integrated and, where feasible, students are pro- 
vided with opportunities to participate in work-based learning. Programs are intended to lead to an 
associate’s or a baccalaureate degree, or a postsecondary certificate in a specific career field, as well as 
placement in appropriate employment or further education [Section 202(a) (3)]. 
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Key 


e How are states and local subgrantees addressing the core elements defined by 


federal legislation? 


The chapter opens with an overview of the four core POS components identified in Per- 
kins IV and highlights Department guidance intended to support states in developing effec- 
tive programs. This section is followed by a description of how states, LEAs, and IHEs 
developed POS and the barriers they encountered in the process. The chapter also discusses 
the extent to which POS created by state agencies and local subgrantees reflect Department 


guidance, as well as state approaches to monitoring and evaluating POS. 


The chapter draws upon several sources of data to answer the research questions. Sutveys 
administered in fall 2009 to all secondary and postsecondary state career and technical edu- 
cation (CTE) directors, a nationally representative sample of local secondary and postsec- 
ondary CTE staff, and the universe of area CTE centers assessed subgrantees’ perceptions 
and opinions on the design and implementation of POS. Case study visits to eligible agencies 
in six states and three local partnerships within each state provided in-depth state and local 
perspectives on developing and implementing POS and the challenges encountered in this 
endeavor. Finally, an expert panel reviewed states’ guidance to LEAs and IHEs on POS de- 


velopment.” 


Information collected for this chapter focuses on state and local activities conducted during 
the 2008-09 program year, the first full year of POS implementation under Perkins IV.3 As 
described in this chapter, many states were still in the early stages of providing assistance to 
local subgrantees in developing and implementing POS. As a result, findings presented do 


not necessarily reflect the current status of POS implementation within states. 


Findings 


1. States and local subgrantees are developing and implementing POS, although the characteristics and 
quality of these programs vary. 


All local providers reported that they were offering POS as of the 2008-09 program year, 
although study data from local site visits revealed considerable variation in the structure and 


quality of POS across states and, in some instances, among local subgrantees within states. 


? See Appendix A for a more detailed description of the study’s methodology. 

3 Although Congress authorized Perkins IV in 2006, states were given a transition year, covering the 
2007-08 program year to develop state plans to respond to new Perkins IV requirements. States’ ap- 
proaches to developing POS varied during this period: some states began program development dur- 
ing the transition period, while others delayed adoption until their state plans were accepted. 
Consequently, data for this section focus on the 2008—09 program year, the first year in which all state 
agencies and local subgrantees were expected to have addressed Perkins IV requirements relating to 
POS design and implementation. 
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This variability is a result of the considerable flexibility that states had in implementing statu- 
tory provisions. For example, most state directors reported that their state neither created 
legislation nor developed administrative guidance to support LEAs and IHEs in developing 
POS. A substantial number of state directors (29 secondary, 20 postsecondary) reported that 
state approval is required for all POS, with both secondary and postsecondary state directors 
reporting that POS developed locally with state guidance were the most common type of 
state-approved POS. 


2. State and local agencies face substantial challenges that complicate POS development and implementa- 


tion. 


State and local administrators reported that they worked in good faith to ensure that each 
Perkins IV subgrantee offered one or more POS. Directors faced a series of challenges in de- 
veloping these programs, most often citing a lack of funds, staff, and time as barriers to POS 
creation. Both secondary and postsecondary state directors also described a lack of under- 
standing among local academic instructors about the purposes of POS, and indicated that 
academic and CTE instructors often lacked sufficient time to plan programs and integrate 
curricula. Local directors reported that shortages of CTE staff and lack of technical expertise 
also were barriers to POS development, though to a lesser extent than other factors. Staff in 
area CTE centers reported difficulty establishing meaningful POS sequences that included 
the academic coursework taken by students in the 9th and 10th grades at their sending 


schools. 


3. Most states and local subgrantees are unable to quantify the number of students enrolled in POS or the 


outcomes that they achieve. 


Although Perkins IV requires that all local subgrantees offer at least one POS, the legislation 
neither offers a statutory definition of a POS student, nor holds state agencies or local sub- 
grantees accountable for reporting student participation rates or educational or employment 
outcomes. As such, relatively few state or local Perkins IV subgtantees are able to produce 
data on the number of students in these programs. Many states and local subgrantees also 
lack the capacity to track secondary students transitioning from high school to postsecond- 
ary education, even when they have established secondary-postsecondary partnerships or 
programs that award postsecondary credit to students while still enrolled in high school. 
State directors reported that, in some instances, they are able to track secondary students in- 
to selected in-state public institutions (e.g., community colleges), but are unable to assess 
student transitions into private two-year or public and private four-year colleges located in or 


out of state. 
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4. Secondary and postsecondary administrators and instructors have difficulty coordinating POS develop- 


ment across levels. 


LEA and IHE program administrators reported that they cooperated to design POS, alt- 
hough case study visits suggested that maintaining communication and fostering collabora- 
tion across the secondary and postsecondary education levels can be logistically challenging. 
Local administrators and staff noted that high school teachers and college faculty often were 
physically isolated from one another, diminishing opportunities to collaborate. Other barri- 
ers included a lack of time and training to support teachers and faculty in aligning secondary 
and postsecondary curricula, the bureaucratic procedures required to develop local articula- 
tion agreements, and resistance from some postsecondary faculty, who considered POS pri- 


marily a secondary program delivery strategy. 


5. The majority of secondary students participating in POS had opportunities to earn college credit, but 
fewer than half of LEA directors reported that their POS led to an industry-recognized credential or a 


postsecondary degree or certificate. 


Perkins IV calls for, but does not require, that secondary students participating in POS have 
the opportunity to earn college credit. Three-quarters of secondary LEA directors reported 
establishing articulation agreements with at least one postsecondary institution, and more 
than half of secondary and postsecondary state directors reported that students in one or 
more of the top five POS offered at the local level could earn dual credit. Although required 
by the legislation, fewer than half of LEA directors reported that their POS led to an indus- 
try-recognized credential or a postsecondary certificate or degree, with many responding that 
they did not know whether these opportunities existed. 


Features of Programs of Study 


Perkins IV requires each local recipient of federal funds to offer one or more POS, and de- 
fines four POS components.* Specifically, a POS should 


e Incorporate secondary and postsecondary education elements; 


e Include coherent and rigorous content aligned with challenging academic stand- 
ards and relevant career and technical content in a coordinated, nonduplicative 
progression of courses that align secondary education with postsecondary edu- 


cation to adequately prepare students for success in postsecondary education; 


4 Sec. 122(c)(1)(A). 
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e Offer the opportunity for secondary education students to participate in dual or 
concurrent enrollment programs or other ways to acquire postsecondary educa- 


tion credits;5 and 


e Lead to an industry-recognized credential or certificate at the postsecondary 


level or an associate’s or bachelor’s degree. 


The legislation requires states to describe how they will consult with LEAs and IHEs to de- 
velop and implement POS in their five-year Perkins IV plans, but the statute provides no ex- 
pectations that states report on the results of this assistance or the status of local 


implementation. 


The POS Design Framework 


To support states in developing POS that incorporate each of the required elements, the 
Department issued a POS Design Framework (Framework) in January 2010 that describes 

10 components considered essential to creating and implementing high-quality comprehen- 
sive POS.° This Department guidance was released nearly three years following the imple- 
mentation of Perkins IV’, meaning that states and local communities worked on POS 
development with limited federal direction. As a consequence, data collected for this study 
document the extent to which state POS development efforts aligned with the essential 
components of effective POS subsequently identified by Department staff. 


> Perkins IV introduces this as an optional component, stipulating that programs “May include the 
opportunity for secondary education students...” to participate in programs that offer postsecondary 
credit. 

6 The Framework currently is posted on the Department’s Perkins Collaborative Resource Network 
(PCRN), a website designed to share resources to promote quality CTE programs and the collection 
of accurate data relating to Perkins IV accountability features. 

http://cte.ed.gov/nationalinitiatives /rposdesignFramework.cfm (accessed October 15, 2010). 
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10. 


POS Design Framework 


Legislation and Polcies—Federal, state, and local legislation or administrative 


policies that promote POS development and implementation. 


Partnerships—Ongoing relationships among education, business, and other 
community stakeholders that are central to POS design, implementation, 


and maintenance. 


Professional Development—Sustained, intensive, and focused opportunities for 


administrators, teachers, and faculty involved in the design, implementation, 


and maintenance of POS. 


Accountability and Evaluation—Systems and strategies to gather quantitative 


and qualitative data on both POS components and student outcomes to aid 


ongoing efforts to develop and implement POS. 


College and Career Readiness Standards—Content standards that define what 
students ate expected to know and be able to do to enter and advance in 


college and/or careers. 


Course Sequences—Nonduplicative sequences of secondary and postsecond- 
aty courses within POS that ensure that students can make a transition to 
postsecondary education without duplicating classes or requiring remedial 


coursework. 


Credit Transfer Agreements—Credit transfer agreements that provide opportu- 


nities for secondary students to gain postsecondary credits, supported by 


formal agreements between secondary and postsecondary partners. 


Guidance Counseling and Advisement—Guidance counseling and advisement 


that help students make informed decisions about which POS to pursue. 


Teaching and Learning Strategies—Innovative and creative instructional ap- 
proaches that enable teachers to integrate academic and technical instruc- 
tion and students to apply academic and technical learning in their POS 


coursework. 


Technical Skill Assessments—National, state, and/or local assessments that 


provide ongoing information on student attainment of the necessary 


knowledge and skills for entry and advancement in postsecondary education 


and careers in their chosen POS. 
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State reactions to department guidance 
State and local staff interviewed during case study visits generally supported the Depart- 
ment’s effort to provide guidance through the POS Framework.’ A majority of state staff 
observed, however, that the guidance would have been helpful earlier in the process and ex- 
pressed frustration at the lack of direction in the initial stages of POS development. One 


state technical assistance provider summed it up in this way: 


It seemed like there wasn’t a complete vision at the federal level as far as 
what they really expected and wanted. Yet they were expecting us to im- 
plement something that they didn’t really know what it was... . It took 
them a long time [to provide examples] and by the time we got them, we’d 


already developed what we thought might work. 
A state director echoed this sentiment: 


The work that was recently done out of [the Department] around POS—I 
wish we would’ve had that three years ago. That should’ve come out imme- 
diately. We wouldn’t have the mess we’re in right now of diversity of states. 
... So it’s going to be more difficult now, because states have already gone 
down certain roads. We’ve got to figure out how to create some national 


structures that make sense. 


Not all state and local providers shared this view. Some believed that an overly prescriptive 
set of requirements would have stifled creativity and innovation, and they appreciated the 
flexibility the Department provided. Moreover, having spent up to three years developing 
and implementing POS on their own, staff in some state agencies expressed concern at the 
prospect of a stricter definition or set of prescriptive regulatory guidance. As one state staff 
member observed, “Having structure and a framework isn’t bad. Requiring us to do certain 


strategies strips the innovation out of it.” 


When guidance was released more than three years after the authorization of Perkins IV, 
some state personnel became wary that the Department would follow with regulatory lan- 
guage that would force educators to modify or reverse what they already had in place. Two 
state-level staff commented that having regulatory guidance would limit collaboration, buy- 
in, and creativity among states, with one noting: “We have close partnerships with our other 
agencies and we work together to figure [POS] out... . If it had been mandated and there 
had been regulations governing it, what we have may not have been as effective as it is be- 


cause there wouldn’t be the ownership that we have now.” 


7 Because the Framework had not yet been released at the time the surveys were conducted, data on 
field perceptions of this guidance are limited to those collected during case study site visits. 
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State staff wondered if states would now be held accountable for all 10 POS components, 

with some expressing mistrust of further regulatory guidance or statutory language. As one 
state director noted, “I think if federal regulations were going to happen for POS, it needed 
to happen at the time it was rolled out. To call it back now and say, ‘Oh, no, you’re doing it 


all wrong.’ I don’t think you can do that...it’s not an acceptable way of doing business.” 


Development of POS 


Percentage of LEAs and IHEs according to whether at least one of their top five POS 


Agency 
LEAs 
IHEs 


Data revealed that states and local subgrantees have developed and implemented POS 
through a variety of processes. In the absence of federal regulatory guidance, some state 
agencies took the lead in establishing statewide POS that were mandatory for local adoption, 
while others opted to provide nonbinding guidance, allowing local subgrantees a measure of 
design flexibility. A handful of states alowed LEAs and IHEs to develop POS on their own 


with little or no state guidance. 


Case study findings suggested that states with strong Tech Prep networks under Perkins LI 
were able to transition to POS with relatively greater ease than those without such systems. 
One state director noted that legislative requirements for POS “gave us the anchor” to rein- 
force and build on work begun under Tech Prep. Survey data supported the case study. 
When reporting on the five POS with the largest student enrollments, 46 percent of LEA re- 
spondents and 56 percent of IHE respondents indicated that one or more POS had previ- 
ously operated as a Tech Prep program (Exhibit 4.1). 


Three states credited “foundational work with Tech Prep” as a very important factor in POS 
development. As one state department of education representative observed, “For a lot of 
the school districts with strong Tech Prep consortiums, POS is a natural and an easy fit.” 
People who had previously been engaged in Tech Prep also took leadership roles in POS de- 
velopment. As one state representative explained, “We’ve had the luxury of having good lo- 
cal people who know how to do Tech Prep. They know how to develop local articulation 


agreements.” 


Exhibit 4.1. 


pteviously was part of a Tech Prep program 


At least one 


None of top five POS Don’t know 
25 46 29 
20 56 25 


Exhibit reads: At the local level, 46 percent of LEAs and 56 percent of IHEs reported that at least one of their top five POS was 
previously part of Tech Prep. 


NOTE: Estimates were weighted to reflect population values. 
SOURCE: LEA and IHE Surveys, 2009. 
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During case study visits, state and local administrators demonstrated varying attitudes toward 
POS. Some identified the POS concept as the most important innovation in Perkins IV, 
crediting it as a lever for reforming CTE. As one state director shared, “It’s a generational 
change here, and we’re hanging POS out there as the banner (for) change” Conversely, oth- 
ers saw it simply as a way of relabeling existing ideas. As one state administrator noted, 
“What’s interesting about all of this talk about POS is it’s not like the concept is new. It’s a 
new title, just as Tech Prep was a new title. But all of the things that were in Tech Prep, and 
all of the things that are in POS are things that we were doing in the 60s.” 


Quality control 
The state approval process helps ensure that POS offered by local providers adhere to a 
minimum level of program quality. In case study interviews, state staff described approval 
processes that involve measuring locally submitted POS against state-established criteria; 
most also said that these processes provided local subgrantees opportunities to revise and re- 
submit their plans for POS should their initial request be denied. When surveyed, 29 second- 
ary and 20 postsecondary state directors reported requiring some form of state approval for 
all POS offered by local subgrantees (Exhibit 4.2). 


Exhibit 4.2. 


183 


Number of states according to the extent to which state approval is required for POS, by 


education level 


Education level Yes, for all Only fora subset Only for one” No 
Secondary 29 6 6 10 
Postsecondary 20 2 8 18 


Exhibit reads: More state directors reported that state approval is required for all POS (29 secondary, 20 postsecondary). Only 10 
secondary and 18 postsecondary state directors reported their states do not require state approval for any POS. 


“The one POS required of each local agency. 
NOTE: N = 51 secondary, 48 postsecondary. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


In contrast, 10 secondary and 18 postsecondary state directors reported that state approval is 
not required for any POS, meaning that the characteristics and components of POS are left 
to local interpretation. One state administrator conceded that this flexibility could lead to 
significant local variations: “We would like to hope there was a typical process for POS de- 


velopment .. . but [there are] a lot of divergent approaches.” 
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POS Implementation: A Case Study Example 


“Hampden” College (name masked to protect anonymity) is a large, urban, four-campus 
community college with several partner school districts. Hampden College’s long- 
standing relationship with its feeder school districts began with the establishment of Tech 
Prep consortia in 1995. The college established 12 POS in the first year (2008-09), and 
was developing 10 more in 2009-10. The college plans to develop about 50 in total, using 


a generic state template as a guide. 


The state recently passed legislation that is intended to increase the rigor of CTE pro- 
grams by integrating them with academic programs. At the secondary level, the legislation 
recommends common planning periods for CTE and academic faculty members, a focus 
on high-wage and high-demand programs, articulation agreements with postsecondary 
institutions, close relationships with business and industry, and cohort scheduling. In re- 
sponse to this legislation, one district’s associate superintendent is enhancing and expand- 
ing CTE by emphasizing career awareness in elementary school and linking middle 


school CTE programs to POS career pathways. 


To fund POS development, the districts and the college agreed to maintain their Tech 
Prep consortium, with each member contributing a percentage of their Perkins funding. 
These funds are used to bring secondary and postsecondary faculty together for Saturday 
workshops that address promoting articulation, integrating academic content into CTE 
course work, and reducing remediation rates for incoming students. For their participa- 
tion, faculty members receive a stipend funded through consortium money. POS devel- 
opment has also benefited from input by longstanding CTE program advisory boards 


made up of alumni and local industry representatives. 


Since 1995, Hampden College has awarded several million dollars to high school students 
in its consortium districts to cover tuition payments for articulated college credits. These 
are included in a student’s transcript upon enrollment at Hampden. At the heart of the 
programs are the articulation agreements for joint assessments: college faculty review 
portfolios of high school students’ work or administer a test, developed in collaboration 
with high school teachers, using the state’s curriculum frameworks as a guide. These ar- 
rangements are made district wide, as well as separately for each program. The college al- 
so offers dual enrollment classes taught at the high school by college faculty or by 
qualified instructors primarily employed by the high school. Students in POS are tracked 
locally through data sharing agreements: districts send coursework information to the 
college, which treats students with qualifying articulated or dual credit as transfer stu- 


dents. 
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Both secondary and postsecondary state directors reported that POS developed locally with 
state guidance were the most common type of state-approved POS. One state identified 
1,748 of this type of POS at the secondary education level, a number far exceeding the na- 
tional median of 84 observed among the 27 states that offered this type of POS (Exhibit 4.3 
and Exhibit 4.4). Within one state, the highest number of state-approved, locally developed 
POS at the postsecondary education level was 452, with the median being 53 among the 16 
states that reported this type of POS.8 


Exhibit 4.3. 
Average and maximum number of state-approved POS developed at the state or local level 


State-approved and State-approved and 
locally developed POS locally developed POS 
State-developed POS with state guidance without state guidance 
Mini- Maxi- Mini- Maxi- Mini- Maxi- 
Education level N Median mum mum N Median mum mum N Median mum mum 
Secondary 29 48 1 200 27 84 1 1,748 5 5 1 144 
Postsecondary 11 16 1 200 16 53 1 452 5 16 1 400 


Exhibit reads: The average number of state-developed POS was 47 for the 29 states with secondary state-developed POS and 34 for the 
11 states with postsecondary state-developed POS. 

NOTE: Of the approved state-developed POS, 7 secondary directors and 3 postsecondary directors indicated the number provided is an 
estimate. For locally developed POS with state guidance, 14 secondary directors and 9 postsecondary directors responded that their 
numbers are estimates. For locally developed POS without state guidance, 1 secondary director and 5 postsecondary directors said their 
numbers are estimates. N = 46 secondary, 34 postsecondary. Exhibit includes only states that reported at least one POS in one of the 
categories. States reporting no POS in any category and states who did not know the number of POS developed across all categories were 
excluded. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


8 Five postsecondary and three secondary directors entered zero across all three categories: state- 
developed POS, locally developed POS with state guidance, and locally developed POS without state 
guidance. These were not included in the average calculations. 
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Exhibit 4.4. 
Number of states according to type of POS development, by education level 


Secondary 


Postsecondary 


Exhibit reads: Thirteen states had only state developed secondary POS; 11 had only locally developed secondary POS with state 
guidance; and 12 had both. Eight states had only state developed postsecondary POS; 13 had only locally developed POS with state 
guidance for postsecondary; and three reported both. Four states had all three types for secondary and no states had all three for 
postsecondary. 

NOTE: n = 46 secondary, 34 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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In contrast to state director reports, 47 percent of LEA directors said that their most com- 
mon state-approved POS were state developed (Exhibit 4.5).° At the postsecondary level, 
the information reported by IHE directors was more consistent with state-level findings: 53 
percent of IHEs identified the most common type of POS as locally developed with state 
guidance. A large percentage of respondents (20 percent or more) also indicated they did not 
know the number or type of POS offered.'° 


Exhibit 4.5. 
Percentage of LEAs and IHEs offering state and locally developed POS, by approval type 


Locally developed Locally developed Locally developed 


POS with state POS without state POS with or without 
State-developed POS guidance guidance guidance 
Approved or will be Approved or will be Approval 
Agency response Approved approved approved not required 
LEAs 
At least one 36! 46! 12! 14! 
None 23 | 24 | 51! 48! 
Don’t know 39! 28 ! 36! 36! 
THEs 
At least one 22! 53 20 ! 20 ! 
None 44! 26! 54! 52! 
Don’t know 33 | 20! 25! 26! 


Exhibit reads: At the local level, 37 percent of LEAs and 22 percent of IHEs offered at least one state-developed POS. Forty-seven 
percent of LEAs and 53 percent of IHEs offered at least one developed locally with state guidance. Only 15 percent of LEAs and 21 
percent of IHEs offered any locally developed POS without state approval. 

! Interpret data with caution. These items have a response tate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


Survey results suggested that IHEs typically offered more state-approved POS options than 
LEAs. LEAs had an average of 10 state-developed POS and eight POS developed locally 
with state guidance (Exhibit 4.6). In contrast, IHEs offered almost twice as many of each 
type of state-approved POS—an average of 18 state-developed POS and 16 locally devel- 
oped POS with state guidance. 


° Because local survey results were not intended to be representative of individual states, care should 
be taken in interpreting this finding. 

10 Zeros were entered across all categories by 104 LEA (1.2 percent) and 37 IHE (3.0 percent) re- 
spondents, indicating that they had no POS of these types. 
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Exhibit 4.6. 
Average and maximum number of POS offered in LEAs and IHEs, by development method 
and approval type 


LEAs IHEs 

Development method and approval type” Average Maximum Average Maximum 
State-developed POS 10.2! 81! 18.1! 128! 
Locally developed POS with state guidance” 8.3! 82! 16.0! 185! 
Locally developed POS without state guidance” 7.0! 107! 17.0! 190! 
Locally developed POS with state guidance: 

Approval not required 6.2! 27! 7.2! 46 | 
Locally developed POS without state guidance: 

Approval not required 7.7 | 83 ! 9.4! 65! 


Exhibit reads: LEAs offered an average of 10 state-developed POS (with a maximum of 81), and IHEs offered an average of 18 state- 
developed POS (with a maximum of 128). 
! Interpret data with caution. These items have a response rate below 85 percent. 


“Each category only includes those cases that had at least one POS reported for that particular category. 


° Only includes those that were locally developed and state-approved. 
NOTE: Estimates were weighted to reflect population values. 
SOURCE: LEA and IHE Surveys, 2009. 


Stakeholder Participation in POS Development 


Because POS curricula must be integrated both vertically (spanning the secondary and post- 
secondary education levels) and horizontally (connecting academic and technical course con- 
tent within levels), secondary and postsecondary stakeholders across academic and CTE 
areas might be expected to be involved in POS development. When surveyed, both second- 
ary and postsecondary state directors reported that they viewed CTE instructors as the 
stakeholder group most involved in POS development (Exhibit 4.7). Specifically, secondary 
state directors reported that secondary CTE teachers (35 states) and postsecondary CTE 
faculty (22 states) participated a lot in POS development, followed by secondary LEA ad- 


ministrators (18 states). 


Similarly, postsecondary state directors reported that postsecondary CTE faculty (24 states), 
secondary CTE teachers (19 states), and postsecondary administrators (16 states) participat- 
ed a lot in POS development. One state director in a case study state credited the Perkins IV 
POS requirement with validating the importance of CTE teachers in educational planning 
because they now “...feel like they’re at the table, they’re accepted more, because they’re 


part of this whole process. Suddenly we’re all on the same chart.” 


In contrast, both secondary and postsecondary state directors viewed academic instructors as 
the least involved among the stakeholder groups. Only five secondary and seven postsec- 


ondary state directors reported that academic instructors at either level participated a lot in 
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POS development. Moreover, many postsecondary state directors reported that they did not 
know whether a particular stakeholder group participated in POS development, suggesting a 


lack of state-level awareness of stakeholder engagement in POS development. 


Exhibit 4.7. 
Number of state directors reporting the extent to which various stakeholders participated in 
POS development, by education level 


Don’t No 
Education level and stakeholder group Not at all Some A lot know response 
Secondary 
Secondary academic teachers 7 35 5 3 1 
Secondary CTE teachers 2 12 35 0 2 
Secondary guidance counselors 8 30 9 3 1 
Secondary LEA administrators 4 27 18 1 1 
National industry, union, and professional groups de) 22 12 3 1 
Postsecondary academic faculty 15 24 7 4 1 
Postsecondary CTE faculty 3 23 22 2: 1 
Postsecondary administrators 8 26 14 2 1 
Local business, unions 7 24 16 3 1 
Local chamber of commetce 19 17 4 10 1 
Postsecondary 
Secondary academic teachers 10 22 5 10 1 
Secondary CTE teachers 7 16 19 5 1 
Secondary guidance counselors 11 24 1 11 1 
Secondary LEA administrators 8 24 6 8 2 
National industry, union, and professional groups 11 24 3 9 1 
Postsecondary academic faculty 6 24 7 10 1 
Postsecondary CTE faculty 2 19 24 2 1 
Postsecondary administrators 3 25 16 3 1 
Local business, unions 4 27 9 7 1 
Local chamber of commetce 19 15 1 12 1 


Exhibit reads: At the state level, 35 secondary CTE directors reported that secondary CTE teachers participated a lot in state agency 
efforts to develop POS, while 24 postsecondaty CTE directors reported that postsecondary CTE faculty did the same. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


A majority (66 percent) of LEA directors reported that secondary CTE teachers had the 
highest level of participation in POS development (Exhibit 4.8). In contrast, only 14 percent 
of LEA directors reported that secondary academic teachers participated a lot in POS devel- 
opment. IHE directors responded similarly, although they reported higher participation of 
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academic faculty than did LEA directors.!! Postsecondary local directors saw secondary 
CTE teachers as participating a lot (37 percent) in POS development, in contrast to only 16 
percent of secondary local directors who saw postsecondary CTE faculty as participating at 
the same level (Exhibit 4.8). Some local directors did not know who participated in POS de- 


velopment from the other level. 


Reflecting the survey findings, the case studies revealed that several local administrators 
identified secondary CTE teachers as important facilitators of POS development. As one 
school superintendent observed, “They’re the ones that drive it really—they’re the ones that 
want to make sure they’re aligned—that their curriculum is aligned, that their kids are pre- 
pated.” Another Perkins IV’ coordinator also attributed progress in POS development to 


“high-energy” teachers with industry knowledge and personal connections at colleges. 


About twice as many IHE (27 percent) as LEA (13 percent) directors reported that local 
businesses participated a lot in POS development. Case studies also revealed that business 
and industry partners participated extensively in POS development in some local communi- 
ties. An administrator in a mid-sized district noted: “If a system does not have strong ties to 
the employer community, then it won’t have valid CTE programs. You can’t have a mean- 
ingful POS unless you... have business and industry coming in and working with students, 


helping you to understand where you’re going and steering you in the right direction.” 


"All comparisons of the IHE and LEA survey data were tested for statistical significance using the 
Student’s #statistic, and all differences cited are statistically significant at the p < .05 level. 
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Exhibit 4.8. 
Percentage of local directors reporting the extent to which various stakeholders participated 
in POS development, by agency 


Agency and stakeholder group Not at all Some A lot Don’t know 

LEAs 
Secondary CTE teachers 4 21 66 8 
LEA administrators 9 41 37 13 
Counselors 14 46 30 11 
Building administrators 12 45 30 14 
Postsecondary CTE instructors 20 40 16 24 
Secondary academic teachers 22 51 14 13 
Local business, unions 16 50 13 ZA. 
Postsecondary administrators 28 35 9 29 
Postsecondary academic instructors 30 35 8 28 
National industry, union groups 34 30 6 31 
Local chamber of commerce 37 26 5 25 

THEs 
Postsecondary CTE instructors 4 28 60 8 
Postsecondary administrators 5 36 52 7 
Secondary CTE teachers 12 38 37 13 
Local business, unions 11 47 27 15 
Postsecondaty academic instructors 20 43 27 10 
LEA administrators 17 30 20 33: 
Advisors 17 43 21 19 
Building administrators 25 29 18 28 
National industry, union groups 27 37 13 24 
Secondary academic teachers 28 41 12 19 
Local chamber of commetce 35 29 6 25 


Exhibit reads: Sixty-six percent of secondary local directors and 60 percent of postsecondary local directors reported that CTE instructors 
in their respective sectors participated a lot in POS development. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


Barriers to POS Development 


Both the surveys and case studies revealed a number of barriers to POS development at the 
state and local levels. Case study findings suggested that building effective partnerships be- 
tween secondary and postsecondary systems was a major hurdle for many states and local 
providers, even though there was widespread understanding that coordination between the 
two levels was a major goal of Perkins IV. The expert panel on POS also found a disconnec- 
tion between the materials provided by secondary and postsecondary state agencies. In the 


sutveys, however, most state and local directors highlighted other factors as more significant 
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impediments to POS development and implementation. The following section provides a 


discussion of the barriers that states, LEAs, and IHEs face. 


State level 


In their survey responses, state directors reported that their most challenging barriers to POS 
development related to their limited capacity to devote staff and fiscal resources to program 
creation. Directors most often cited shortages of federal funding, state staff, and time as the 
greatest challenges to POS development (Exhibit 4.9 and Exhibit 4.10). Both secondary and 
postsecondary state directors also cited a lack of understanding of POS among local academ- 
ic instructors as one of the biggest barriers to POS development. Additionally, postsecond- 


ary state directors cited a lack of influence among CTE staff as a barrier. 


that various barriers limited POS development, by type of barrier 


Barrier 


Academic teachers do not have a good 
understanding of POS 


Inadequate federal funding 

Shortage of state staff 

State staff lack adequate time 

Little state staff influence at local level 

CTE teachers do not understand POS 

Lack of skills or time for curriculum integration 
Lack of statewide technical content standards 
Lack of technical skills assessments 


Inadequate federal guidance 


Lack of statewide curricular materials 


State staff lack technical expertise in POS design 


Exhibit 4.9. 
Number of secondary state directors according to the extent to which they agreed 


Strongly 
disagree Disagree opinion 
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Exhibit reads: Among secondary state CTE directors, 23 strongly agreed and 17 agreed that secondary academic teachers did not have a 


good understanding of POS, which was a barrier to POS development. 


NOTE: N = 51. 
SOURCE: Secondary State Director Survey, 2009. 
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Exhibit 4.10. 


Number of postsecondary state directors according to the extent to which they agreed that 


various barriers limited POS development, by type of barrier 


Not 
Strongly No Strongly applic- No 

Barrier disagree Disagree opinion Agree agree able response 
State staff lack adequate time 3 11 0 13 17 3 1 
Shortage of state staff 2 6 5 13 15 6 1 
Inadequate federal funding 3 6 3) 16 15 2 1 
Academic faculty do not have a good 

understanding of POS 2 3 11 13 15 2 2 
Little state staff influence at local level 4 14 2 16 9 2 
CTE faculty do not understand POS 6 11 5 14 9 2 
Lack of technical skills assessments 12 11 6 8 8 2 
Lack of skills or time for curriculum integration 2 9 16 10 8 2 
Lack of statewide curticular materials 7 16 7 6 7 3 2 
Lack of statewide technical content standards 11 12 6 7 6 5 
Inadequate federal guidance 7 11 4 17 5 6) 
State staff lack technical expertise in POS design 14 13 2) 7 4 4 


Exhibit reads: Among postsecondary state CTE directors, 17 strongly agreed and 13 agreed that state staff did not have adequate time to 
support POS development. 

NOTE: N = 48. 

SOURCE: Postsecondary State Director Survey, 2009. 


While state directors at both levels often cited lack of staff time and other resources as barri- 
ers, they generally had a positive view of cooperation between secondary and postsecondary 
state staff on POS development. A majority of state directors agreed or strongly agreed (32 
secondary, 33 postsecondary) that state staff cooperated in developing POS (Exhibit 4.11). 
They also perceived high levels of cooperation between secondary and postsecondary local 
administrators: 28 secondary and 28 postsecondary state directors agreed or strongly agreed 


that secondary and postsecondary local administrators cooperate to create POS. 


Given the long-standing focus on academic and technical integration in Perkins, it might be 
expected that collaboration between academic and CTE instructors would be relatively 
strong. However, a number of state directors indicated that this was not the case, and sec- 
ondary state directors were nearly twice as likely as their postsecondary colleagues to express 
concern about the level of cooperation between academic and CTE instructors. At the sec- 
ondary level, 17 state directors disagreed with the statement that secondary CTE and aca- 
demic teachers cooperate to create POS, while only eight postsecondary state directors 


disagreed. 
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Exhibit 4.11. 
Number of state directors according to the extent to which they agreed or disagreed that 
state and local staff cooperated on POS development, by education level 


Not 
Strongly No Strongly — applic- No 
Education level and staff cooperating disagree Disagree opinion Agree agree able response 
Secondary 
Secondary CTE and academic teachers 
in LEAs cooperate to create POS 3 17 10 9 8 3 
Secondary and postsecondary instructors 
cooperate to create POS 1 6 6 23 11 3 1 
Secondary and postsecondary administrators 
cooperate to create POS 3 11 6 17 11 2 
Secondary and postsecondary state staff 
cooperate to implement POS 2 5 5 15 17 6 1 
Postsecondary 
CTE and academic teachers in IHEs 
cooperate to create POS 3 8 10 22 3 1 
Secondary and postsecondary instructors 
cooperate to create POS 2 4 6 28 2 2 1 
Secondary and postsecondary local 
administrators cooperate to create POS 3 8 7 20 8 1 
Secondary and postsecondary state staff 
cooperate to implement POS 1 6 4 16 17 2 


Exhibit reads: Seventeen secondary and 25 postsecondary state directors agreed or strongly agreed that CTE and academic instructors in 
their respective sectors cooperate to create POS. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Local level 
According to both LEA and IHE directors, perceived shortages of funds and time posed 
challenges to POS development at the local level (Exhibit 4.12). Almost half of local direc- 
tors at both levels (between 44 and 48 percent) reported that these shortages limited their ef- 
forts to develop POS a lot, while an additional third (between 33 and 35 percent) reported 
some negative effect. Respondents also reported that shortages of local CTE staff and lack 


of technical expertise were barriers to local POS development, though to a lesser extent. 
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Exhibit 4.12. 
Percentage of LEAs and IHEs according to the extent to which various barriers 
limited POS development, by type of barrier 
Agency and barrier Not at all Some A lot Don’t know 
LEAs 
Shortage of time 11 33 48 8 
Shortage of funds 14 33 45 9 
Shortage of LEA CTE staff 36 36 18 10 
Lack of technical expertise 46 35 6 12 
Resistance to state influence at local level 60 21 4 16 
Conflicts between relevant individuals 55 25 4 16 
Lack of cooperation by state level staff 68 13 3 17 
IHEs 
Shortage of time 12 35 48 5 
Shortage of funds 17 33 44 6 
Shortage of institutional CTE staff 35 43 15 7 
Lack of technical expertise 53 35 5 8 
Resistance to state influence at local level 60 23 4 13 
Conflicts between relevant individuals 59 26 2 13 
Lack of cooperation by state level staff 72 13 2 12 


Exhibit reads: At the local level, 48 percent of secondary and 48 percent of postsecondary local directors reported that shortage of time 


limited their efforts to develop POS a lot. 


NOTE: Estimates were weighted to reflect population values. 


SOURCE: LEA and IHE Surveys, 2009. 


Case study data reinforced these survey findings, revealing that pervasive budget shortfalls 
caused state administrators and local educators to worry more about program survival than 
about POS development. As a staff member at one state agency noted, “We could not sur- 
vive without Perkins IV funding. Our program survives because of the small amount of 
money that we have from Perkins...we do the best we can.” Similarly, one local postsecond- 
ary administrator explained the time constraints he experienced with POS development. 
“Tve got a full-time job as a dean, and I took this on. It’s difficult to spend the amount of 
time it would take to really do it right.” A postsecondary administrator in another state also 
cited time, or lack of it, as a primary barrier to POS development and implementation, 
commenting that it takes “...a lot of time to meet with secondary partners to align course 
sequences. I think part of it is just the newness of it, and I think it’s easier now but it’s still a 


challenge.” 
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Sixty-eight percent of secondary and 72 percent of postsecondary local directors reported 
that lack of cooperation by state-level staff was not at all a barrier to POS development. In 
addition, about half of local directors at both levels (between 55 and 60 percent) said that 
conflicts between relevant individuals and resistance to state influence were not barriers to 
POS development at all. 


In contrast to the survey responses, case study findings suggested that educators face chal- 
lenges in forming effective secondary-postsecondary partnerships. There were significant 
variations among the number and effectiveness of secondary-postsecondary partnerships 
across districts and between comprehensive high schools and area CTE centers. Success ap- 
peared largely dependent on relationships between representatives of the two types of insti- 
tutions at the local level and the commitment of individual administrators or faculty 


members. 


Local administrators frequently cited the enormous effort required to forge secondary- 
postsecondary connections and the lack of time and resources for both partners as barriers 
to POS development. For example, several days may be required to walk faculty through the 
steps and challenges of aligning secondary and postsecondary curricula, the step-by-step 
process of developing a local articulation agreement and, in the words of one state official, 
overcoming the “preconceived notions [about the unpreparedness of secondary students] 
that exist sometimes on the part of the postsecondary faculty.” Finding sufficient resources 


and staff release time can be a challenge to conducting such trainings. 


In five of the six case study states, respondents described the particular challenges that small, 
rural school districts in remote areas face in developing POS. In particular, they said that ru- 
ral school districts often lack the staff to offer a full sequence of POS courses spanning sec- 
ondary and postsecondary education. A representative of one small district described the 
difficulty of applying the state-designed POS Framework locally due to limited course offer- 
ings and instructional capacity. A number of local secondary practitioners also cited difficul- 
ties in finding local postsecondary partners. In three of the states visited, rural districts 
reported that they were unable to access postsecondary partners regularly because of dis- 
tance and, in some circumstances, inclement weather or hazardous road conditions. As a 
secondary CTE director in a rural district said, “I meet the requirements for having a POS, 
but it is aligned with a college 90 miles down the road in a different county.” In addition, 
some small rural districts reported difficulty in finding appropriate business and community 


representatives to serve on advisory committees. 
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Enrollment in POS 


Perkins IV does not require state agencies or local subgrantees to report on student participa- 
tion or to differentiate POS outcomes within their accountability measures. In the absence 
of federal reporting requirements, few state and local practitioners reported that they collect 
POS enrollment or outcome data. At the state level, 13 secondary and 21 postsecondary di- 
rectors reported that they did not know how many CTE students in their state were enrolled 
in POS (Exhibit 4.13). Case study interviewees often hedged or protested outright when 
asked about enrollments, with officials in two states characterizing reporting on POS out- 
comes as an unfunded mandate. As one official explained, “Had that been built into Per- 
kins IV legislation, then we would’ve looked at it differently. But it wasn’t, and so we just 
followed the language of the law.” Officials in only one of six case study states reported hav- 
ing the capacity to distinguish POS students at the secondary level from the broader CTE 
population. Moreover, only one case study state had a longitudinal data system in place for 
tracking K—12 students, but officials reported that they could not use this system, as current- 
ly configured, to identify POS participants. 


Exhibit 4.13. 
Number of states according to the percentage of CTE students enrolled in POS coursework 
in 2008-09, by education level 


Less than Between Between Between Don’t No 
Education level None 25% 26-50% 51-75% 76-99% All know response 
Secondary 4 7 5 7 6 8 13 1 
Postsecondary 6 8 4 1 3 4 21 1 


Exhibit reads: Eight secondary and four postsecondary state directors reported that all students participated in POS coursework in 
2008-09. Many state directors (13 secondary, 21 postsecondary) did not know if CTE students participated in POS coursework during that 
year. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Case study states also varied in their capacity to follow POS students as they transitioned 
from secondary to postsecondary education. As one state official noted, “We currently don’t 
have an easy way at either education level office [secondary or postsecondary] to watch stu- 
dents move through one POS segment to another.” Case study respondents in three states 
cited the Family Educational Rights and Privacy Act (FERPA) or state restrictions on the use of 
Social Security numbers (SSN) as barriers to tracking students across secondary and postsec- 
ondary levels. They suggested that a common, unique identifier would facilitate tracking stu- 
dent transitions. However, following students across education levels was complicated even 
in those states with identifiers and mechanisms in place to track students. For example, only 
one of three states that had access to the National Student Clearinghouse reported having 


the capacity to follow secondary students into postsecondary education and beyond. 
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Officials in one state questioned the utility of tracking secondary POS students into postsec- 
ondary institutions as a measure of success, since a secondary POS student who learned that 
the POS career pathway was not for them experienced a valuable outcome. Several local re- 
spondents made similar observations about the futility of tracking students: one noted that 
students who leave a program early to work or who attend multiple colleges to complete 


their training would not be included in counts of successful POS completers. 


Both secondary and postsecondary state directors reported that Business Management and 
Administration, Information Technology, and Architecture and Construction were in the top 
five POS (by Career Cluster!) in terms of enrollment (Exhibit 4.14). Secondary state direc- 
tors also identified Human Services and Agriculture, Food, and Natural Resources among 
the POS with the highest enrollments in the state, while postsecondary state directors cited 


Manufacturing and Health Sciences. 


'2 Career clusters are an organizing tool to promote seamless student transitions from education to ca- 
reer. The States’ Career Clusters Initiative, established and maintained by the National Career Tech- 
nical Education Foundation, is intended to help states connect CTE to education, workforce 
preparation, and economic development. For more information on the initiative, refer to the initiative 
website: http://www.careerclusters.org/. 

13'The LEA and IHE surveys asked local directors to list the five POS with the highest enrollments in 
their district or institution. A subsequent set of questions referred respondents back to their list of the 
five POS with the highest enrollments (hereafter referred to as “top five POS”). 
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Exhibit 4.14. 
POS with the highest enrollments according to state directors who reported the top five POS 
with the highest enrollments in their state, by education level and Career Cluster 


Number of states 


Number of listing in top five 
states reporting in Avetage that don’t know 
Education level and Career Cluster* top five enrollment exact enrollment 
Secondary 
Business Management and Administration 23 10,368 ? 8 
Human Services 23 13,779 © 9 
Agriculture, Food, and Natural Resources 22 8,716 41-4 
Architecture and Construction 18 15,285 © 9 
Information Technology 16 2,426 : g& 
Postsecondary 
Health Sciences 25 13,889 " 12' 
Business Management and Administration 24 26,838 10 
Manufacturing 14 3.335 . 4 
Architecture and Construction 11 2,819 8 
Information Technology 11 1,956 4 


Exhibit reads: The Business Management and Administration Cluster was one of the top five secondary POS with the highest enrollments 
in 23 states, with an average enrollment of 10,368 students. The Health Sciences Cluster was one of the top five postsecondary POS with 
highest enrollments in 25 states, with an average enrollment of 13,889 students. 

“Survey respondents wrote in their five POS with highest enrollments and then selected the Career Cluster under which each POS was 


categorized. The identified clusters were used for this analysis. 
b 


= 


Three of the states had two of their top five POS in the Business Management and Administration Cluster. The sum of all enrollment 
numbers was used for calculating the average; however, the sum was taken as the average across eight states because only eight were unique 
respondents (not 11). 


“ Three of the 14 respondents had two of their top five POS in the Human Services Cluster. The average does not include an outlier of 
three. 

* One of the 11 respondents had four of their top five POS in the Agriculture, Food, and Natural Resources Cluster and did not know 
enrollment numbers for any. Another respondent had two of their top five POS in the Agriculture Cluster, but did not know enrollment 
numbers for either. 

“ The average does not include an outlier of five. 

"One of the 16 respondents for this question had two of the top five POS in the Information Technology Cluster. The average does not 
include an outlier of 32,954. 

® Includes one state who had two of its top five POS in the Information Technology Cluster, but did not know enrollment numbers for 
either. 

" One state had three of its top five POS in the Health Sciences Cluster. 

"Ten unique responses (two states who had two of their top five POS in the Health Sciences Cluster did not know enrollment numbers for 
either). 

i Three states had two of their top five POS in the Business Management and Administration Cluster. 

“ One state had three of its top five POS in the Manufacturing Cluster. The average does not include an outlier of 38,000. 

NOTE: Percentages are calculated from the total number of state directors at each level (not the total who responded to the question). The 
average enrollments are based only on states that reported enrollment numbers. States that did not provide responses were not included in 
calculations of average enrollment. Exhibit includes response only from states that responded to the survey question. States that did not 
respond are excluded. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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According to secondary local directors, the POS with the highest LEA enrollments were Ag- 
ticulture, Food, and Natural Resources; Business Management and Administration; Health 
Sciences; Architecture and Construction; and Human Services (Exhibit 4.15).'4 In contrast, 
the POS with the highest enrollments for postsecondary education were Health Sciences; 
Business Management and Administration; Information Technology; Transportation, Distti- 


bution, and Logistics; and Education and Training. 


Exhibit 4.15. 
Top five POS with highest enrollments according to local directors, by agency and Career 
Cluster 
Agency and Career Cluster Percent of local directors 
LEAs 
Agriculture, Food, and Natural Resources 43! 
Business Management and Administration 38 ! 
Health Sciences 28 ! 
Architecture and Construction 25! 
Human Services 24! 
IHEs 
Health Sciences 52! 
Business Management and Administration 38! 
Information Technology 26! 
Transportation, Distribution, and Logistics 21! 
Education and Training 18! 


Exhibit reads: The Agriculture, Food, and Natural Resources Cluster was a top five POS for 43 percent of LEAs, while Health Sciences 
was a top five POS for nearly 53 percent of IHEs. 


! Interpret data with caution. These items have a response rate below 85 percent. 


“Survey respondents wrote in their five POS with highest enrollments and then selected the Career Cluster under which each was 
categorized. The identified clusters were used for this analysis. 


t ~ , : , ; 
° Number of entities reporting this Career Cluster in their top five. 


NOTE: Estimates were weighted to reflect population values. 
SOURCE: LEA and IHE Surveys, 2009. 


Plans for POS Expansion 


Perkins IV requires each LEA and IHE to offer at least one POS. Findings from case studies 


and surveys suggested that subgrantees’ commitment to POS implementation and expansion 


14 Exhibit 4.15 shows the sum of POS developed in each Career Cluster for the top five Clusters, as 
well as the percentage of local agencies reporting the particular Cluster in their top five. 

15 Data were drawn from national samples at each level rather than using matched pairs of LEAs and 
IHEs. Therefore, these data do not allow an analysis of whether secondary POS students who enroll 
in postsecondary education continue in the same program. 
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varies significantly. As one state director explained, “My gut tells me that we’re at the com- 
pliance level with some people [exceeding] that. But most people are probably at the compli- 


ance level.” 


Many states did not have policies that require all CTE programs to eventually become POS. 
Mote states, however, had such a policy at the secondary level (28) than at the postsecondary 
level (15), although the majority of these states did not have a specific timeline for the transi- 
tion (19 secondary, 10 postsecondary) (Exhibit 4.16). 


Exhibit 4.16. 
Number of states according to whether state administrative policy requires all CTE 
programs to eventually become POS, by education level 


Education level Yes, duting Perkins IV lifetime Yes, no timeline No Don’t know 
Secondary 9 19 23 0 
Postsecondary 5 10 32 1 


Exhibit reads: Many states did not have administrative policies that require all CTE programs to eventually become POS (23 secondary, 32 
postsecondary). 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Nonetheless, few case study respondents viewed POS as a temporary phenomenon. Admin- 
istrators and staff at both the state and local levels discussed at length their plans for expand- 
ing POS offerings in the immediate future and over the long term. Plans for expansion were 
guided by a variety of factors: industry needs, labor market conditions and trends, student 
cateet choices, and the interests of local administrators and faculty members. One local ad- 
ministrator observed how valuable it would be if CTE teachers used POS more extensively 
as part of their career planning units, if school counseling staff used POS to advise students 
on different career paths, and if administrators used the POS concept to organize their 


thinking about the importance of both college and careers for students. 


Some local administrators indicated that the implementation of POS, notably the integration 
of CTE and academic courses, was changing perceptions about CTE and CTE students 
themselves. As one local director stated, “No longer is CTE seen as a dumping ground, but 
mote of a proving ground.” Officials in one state reported using the POS concept to market 
CTE to a broader segment of secondary students: “Part of the message that we bring is the 
fact that this [POS] is about college and career readiness for all students.” Officials in other 
states, however, still found it challenging to promote the value of CTE. As one veteran state 
staffer lamented, “What we haven’t managed to really do is sell this idea that POS is the way 
to do it... to deliver CTE. That is the way.” 
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Design of POS 


As noted previously, Perkins IV identifies four core elements that broadly define a POS. To 
support states in developing effective POS that address these elements, the Department sub- 
sequently released a design framework that incorporates these elements as well as supporting 
components that should be considered in the design and implementation of POS. The fol- 
lowing sections describe how state and local practitioners are addressing the core elements 


and supporting components in their efforts to create and deliver POS. 


Perkins IV Core Elements 


New statutory language in Perkins IV requires eligible agencies to describe POS that may be 
adopted by LEAs and IHEs as an option for students to use in planning and completing fu- 
ture coursework in CTE content areas. These descriptions, which were to be included in 
each state’s Perkins plan, were expected to encompass the four core elements specified in the 
legislation. Perkins IV also indicated that state plans should include information about how 
eligible agencies would develop and implement POS, as well as support LEAs and IHEs in 
carrying out specified activities. Findings related to states’ efforts to address each of the four 


elements are presented below. 


Core element one: Secondary-postsecondary elements 
The first core element of Perkins IV requires that POS include elements from both second- 
ary and postsecondary education. Survey responses indicated that state officials often did not 
require, and local CTE administrators often did not include, secondary and postsecondary 
elements in their POS. At the state level, only 28 secondary and 20 postsecondary directors 
reported that state-approved POS spanned secondary and postsecondary education in their 
state (Exhibit 4.17). At the local level, 68 percent of secondary and 70 percent of postsec- 
ondary local directors reported that at least one of the five POS with the largest enrollments 
spanned both education levels (Exhibit 4.18 and Exhibit 4.19). 
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Exhibit 4.17. 


Number of states according to various curriculum requirements for POS, by education level 


Education level and curriculum requirement Required Not required Don’tknow No response 
Secondary 
Spans secondary and postsecondary 28 22 0 1 
Nonduplicative across secondary and postsecondary 32 13 4 2 


POS uses curriculum that is... 


Locally developed 22 25 1 3 
State-developed 18 30 1 2 
Industry-developed 12 33 2 4 
Third patty (vendor)-developed 2. 43 2 4 
Other 9 14 4 24 
Postsecondary 

Spans secondary and postsecondary 20 25 0 

Nonduplicative across secondary and postsecondary 23 23 0 2 

POS uses curriculum that is... 
Locally developed 25 16 3 4 
State-developed 6 36 4 2 
Industry-developed 11 29 5 3 
Third party (vendor)-developed 1 40 4 3 
Other 5 12 5 26 


Exhibit reads: More than half of secondary state directors reported that POS are required to span secondary and postsecondary levels 
(28), while fewer postsecondary directors reported the same (20). States were most likely to require state-approved POS to use locally 
developed curriculum for both secondary (22) and postsecondary (25). 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Exhibit 4.18. 
Percentage of LEAs reporting the use of various curriculum characteristics in at least one of 
their top five POS 


At least one of 


Curriculum characteristic None top five POS Don’t know 
Spans secondary and postsecondary 8 68 24 
Nonduplicative across secondary and postsecondary 16 42 42 


POS uses curriculum that is... 


State-developed 18 65 17 
Industry-developed 22 48 30 
Third party (vendor)-developed 42 23 35 
Locally developed 26 53: 21 
Other 43 | 4! 52! 


Exhibit reads: Almost 68 percent of LEA directors reported that at least one of their top five POS spanned the secondary and 
postsecondary levels, but only 42 percent reported that at least one of their top five POS was nonduplicative across levels. About 65 
percent of LEA directors reported that at least one of their top five POS used state-developed curriculum. 

! Interpret data with caution. These items have a response rate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA Survey, 2009. 


Exhibit 4.19. 
Percentage of IHEs reporting the use of various curriculum characteristics in at least one of 
their top five POS 


At least one of 


Curriculum characteristic None top five POS Don’t know 
Spans secondary and postsecondary LS: 70 15 
Nonduplicative across secondary and postsecondary 16 59 25 


POS uses curriculum that is... 


State-developed 31 55 15 
Industry-developed 25 58 16 
Third party (vendor)-developed 62 14 24 
Locally developed 20 66 15 
Other 50! 9! 40! 


Exhibit reads: More than half of IHE directors reported that at least one of their top five POS spanned the secondary and postsecondary 
levels (70 percent) and was nonduplicative across levels (59 percent). The majority (66 percent) reported that at least one of their top five 
POS used locally developed curriculum. 

! Interpret data with caution. These items have a response rate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: IHE Survey, 2009. 


CHAPTER 4. PROGRAMS OF STUDY 205 


Case study respondents noted challenges in creating the necessary secondary-postsecondary 
partnerships for POS. While educators seemed to understand that POS are meant to incor- 
porate secondary and postsecondary elements, POS development activities tended to con- 
centrate on the secondary level. Staff at one state agency suspected that some secondary 
subgrantees had completed the postsecondary portion of the statewide POS template simply 
by looking at their local community college’s course catalog and filling in plausible related 
coursework. Indeed, the case study visit to a rural district in the state confirmed this observa- 


tion. 


The expert panel convened to review state POS materials also noted an absence of state 
guidance to support the alignment of POS with the postsecondary level. The panel reported 
that states’ POS guidance focused heavily on the design and implementation of POS at the 
secondary level, with relatively little attention paid to the unique characteristics of postsec- 


ondary educational programs. 


The case study team also observed instances of alignment between the secondary and post- 
secondary POS elements. For example, local staff in several sites identified cooperation be- 
tween secondary and postsecondary institutions as key to successful POS development and 
implementation. As one postsecondary local director explained regarding course sequences, 
“Having those meetings, where we sat together [with secondary colleagues] and worked 
them out, helped us to prepare POS locally.” Similarly, in explaining how his program was 
related to a local postsecondary institution, a secondary local director said, “Early on, way 
early on, it was about [secondary instructors and postsecondary faculty] getting to know each 
other, getting to understand this [POS] process if we’re going to work together. ... We really 


are a team to make good things happen for students.” 


In some local communities, efforts by secondary staff and teachers to work with postsec- 
ondary partners on POS were crucial to forging relationships across education levels and de- 
veloping strong programs. According to a senior state staff member, educators in many 
school districts “see the benefit of programs of study now. A lot are venturing out. They’re 
being very innovative with their CTE programs, and they’re looking for colleges that will ar- 
ticulate some of the programs with them.” On the other end of the spectrum, some sites 
were not actively seeking partnerships; in one local case study site, for example, the second- 
ary Perkins IV coordinator could not even identify her counterpart at the local community 


college. 
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Core element two: Academic and technical rigor 
Perkins IV requires that POS strive to develop the academic and technical skills of secondary 
and postsecondary students enrolling in CTE coursework. To do so, the legislation builds 
upon and promotes long-standing efforts by states and localities to create and integrate chal- 
lenging academic and technical standards into CTE curricula. States have considerable flexi- 
bility in determining the scope, sequence, and rigor of the curricula and standards they create 


ot adopt, as highlighted below. 


Curriculum 
About half of states required state-approved POS to be aligned with locally developed cur- 
riculum (22 secondary, 25 postsecondary). Fewer states required alignment with state- 
developed, industry-developed, or some other curriculum type (Exhibit 4.17). Consistent 
with state-level findings, half (53 percent) of secondary local directors reported that at least 
one of their top five POS incorporated locally developed curricula, and 65 percent reported 
alionment with state-developed curricula (Exhibit 4.18). About two-thirds (66 percent) of 
postsecondary local directors reported that at least one of their top five POS incorporated 
locally developed curricula (Exhibit 4.19). 


Although Perkins IV clearly indicates that POS courses should be aligned and nonduplicative 
across secondary and postsecondary education, the survey findings revealed that not all POS 
met this requirement. At the state level, only 32 secondary and 23 postsecondary directors 
reported that their states required state-approved POS coursework to be nonduplicative 
across education levels (Exhibit 4.17). At the local level, only 42 percent of secondary and 59 
percent of postsecondary local directors reported that at least one of their top five POS was 
nonduplicative across both education levels (Exhibit 4.18 and Exhibit 4.19). About 42 per- 
cent of secondary and 25 percent of postsecondary local directors reported that they did not 


know if any of their top five POS were nonduplicative. 


Standards 

States reported that CTE programs reflected a variety of content standards, including those 
developed by the States’ Career Clusters Initiative, industry-based standards, and state- or lo- 
cally developed standards, and many reported using more than one. Secondary state directors 
reported that the most common standards used were state developed (41 states), industry 
developed (38 states), or based on the Career Clusters (24 states) (Exhibit 4.20). Postsecond- 
ary state directors reported using standards developed in consultation with a business or ad- 
visory council (33 states) and standards adopted from industry groups by local instructors 


(24 states) most frequently. 
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Exhibit 4.20. 
Number of states using various approaches to developing content standards for 


state-approved POS, by education level 


Standard Secondary Postsecondary 
State develops secondary CTE standards 41 n/a 
State develops postsecondary CTE standards n/a 10 
State adopts industry-based standards 38 17 
State adopts 16 Career Cluster standards 24 14 
Local instructors develop content standards based on state criteria 19 14 
Local instructors develop their own standards 8 17 
Local instructors adopt standards created by industry groups 19 24 
Local instructors consult with business or advisory council to develop or select stand- 

ards 26 33 
No standards exist 1 Do 
Don’t know 0 0 
No response 1 eS) 


Exhibit reads: Forty-one secondary state directors reported that the state developed secondary CTE content standards for state- 
approved POS, while 33 postsecondary state directors reported that local instructors consulted with business or advisory councils to 
develop or select content standards for state-approved POS. 

n/a = Not available. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


The surveys asked state and local directors about the extent to which state-approved POS 
were aligned with CTE standards. Although coherent POS that span both secondary and 
postsecondary education should in theory have a sequenced, single set of standards, survey 
responses differed significantly across the two levels. According to state directors, more sec- 
ondary state POS requirements included state and national standards, whereas mote post- 
secondary state POS requirements included local standards. Among state directors, 35 
secondary and 25 postsecondary directors reported that their states required state-approved 
POS to be aligned with industry-developed technical standards (Exhibit 4.21). Many states 
also required that state-approved POS be aligned with state-developed technical standards 
(31) and Career Clusters standards (29) for secondary education. Meanwhile, half of postsec- 
ondary state directors reported that their states required state-approved POS to be aligned 
with locally developed technical standards. ' 


16 Responses were not mutually exclusive; numbers reflect the number of respondents for each op- 
tion. 
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Exhibit 4.21. 
Number of states according to required alignment of state-approved POS with various types 
of standards, by education level 


Not Don’t No 

Standard Required required know response 
Secondary 

State postsecondary standards or program completion requirements 33 15 2 1 

State-developed technical standards 31 17 1 2 

Industry-developed technical standards 35 14 0 2 

Technical standards based on Career Clusters 29 16 1 5 

Locally developed technical standards 16 31 1 3 

Other technical standards 9 10 6 26 
Postsecondary 

State postsecondary standards or program completion requirements n/a n/a n/a n/a 

State-developed technical standards 18 26 2 2 

Industry-developed technical standards 25 16 4 3 

Technical standards based on Career Clusters 19 21 3 

Locally developed technical standards 25 17 3 3 

Other technical standards . 9 8 26 


Exhibit reads: The majority of states (35) required LEAs to use industry-developed standards for state-approved POS. Half of states 
required IHEs to use industry-developed and locally developed technical standards. 

n/a = Not available. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


The responses of local directors did not precisely match those of state directors. While sur- 
vey findings suggested that the majority (81 percent) of LEAs were aligning at least one of 
their top five POS with state secondary academic standards, only about half (57 percent) 
aligned their POS with industry-developed technical standards. Between 16 and 33 percent 
of LEA respondents, depending on the standard, reported not knowing whether their top 
five POS were aligned with a particular standard type (Exhibit 4.22).1” 


17 Tn many instances throughout the survey, secondary and postsecondary state and local directors in- 
dicated that they did not have the information needed to answer the question. While this may indicate 
a lack of awareness on the part of some directors, it also may indicate the complexity associated with 
developing and implementing POS statewide. Given that many state approved POS are locally devel- 
oped, the information needed to answer survey questions may not be routinely communicated be- 
tween state and local directors. For this reason, one should be cautious when reviewing the survey 
findings. 
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Exhibit 4.22. 
Percentage of LEAs according to the inclusion of various standards in at least one of their 
top five POS 
At least one 

Standard None of top five POS Don’t know 
State secondary academic standards 3 81 16 
Postsecondary standards or requirements for program completion 4 73 23 
State-developed technical standards 6 74 21 
Industry-developed technical standards 11 57 33 
National technical standards (Career Clusters) 9 60 31 
Locally developed technical standards 24 49 27 


Exhibit reads: Nearly 81 percent of LEAs incorporated state secondary academic standards in at least one of their top five POS. 
NOTE: Estimates were weighted to reflect population values. 
SOURCE: LEA Survey, 2009. 


Among IHEs, 81 percent incorporated state postsecondary standards or requirements for 
program completion in at least one of their top five POS (Exhibit 4.23). About two-thirds of 
IHEs incorporated state-developed technical standards (67 percent) and industry-developed 
technical standards (71 percent) in at least one of their top five POS. [HE directors respond- 
ed more often than LEA directors that “none” of their POS aligned with the standards 


listed. 
Exhibit 4.23. 
Percentage of IHEs according to the inclusion of various standards in at least one of their 
top five POS 
At least one of 

Standard None top five POS Don’t know 
State postsecondary standards or requirements for program completion 6 81 13 
State-developed technical standards 14 67 19 
Industry-developed technical standards 11 71 17 
National technical standards (Career Clusters) 18 50 32 
Locally developed technical standards 22 60 18 


Exhibit reads: The majority of IHEs aligned at least one of their top five POS with state postsecondary standards or requirements for 
program completion (81 percent). More than half aligned at least one of their top five POS with state-developed (67 percent) or industry- 
developed technical standards (71 percent). 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: IHE Survey, 2009. 
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Core element three: Dual enrollment 
Perkins IV requires that states provide high school students enrolling in POS with the oppor- 
tunity to participate in dual enrollment, though the provision is offered as an option, not a 
requirement. Specifically, the legislation directs that POS “May include the opportunity for 
secondary education students to participate in dual or concurrent enrollment programs or 
other ways to acquire postsecondary education credits” (Sec. 122(c)(1)(A). Noting the ambi- 
guity of this guidance, a postsecondary administrator in one case study state commented, 
“T’m not sure if dual enrollment is actually part of the POS. It’s very confusing what is and is 
not part of the POS.” There also is some uncertainty within states on terminology. Second- 
ary and postsecondary educators in many case study sites used the terms “dual enrollment,” 
“articulation,” and “dual credit” —or some combination thereof—interchangeably and in 
many cases, they were not sure of how each of these options was precisely defined in the 


state. 


Irrespective of how the agreements were labeled, many state and local agencies had estab- 
lished articulation or dual enrollment agreements that enabled secondary students to earn 
postsecondary credit. Some local case study sites had maintained articulation agreements es- 
tablished under Tech Prep, while others had enhanced or transformed them into dual credit 
or dual enrollment arrangements. About half (28 secondary, 25 postsecondary) of state direc- 
tors reported that state-approved POS were required to have an articulation agreement be- 
tween secondary and postsecondary education (Exhibit 4.24). However, only 21 
postsecondary state directors reported that they required state-approved POS to offer sec- 


ondary CTE courses that allow students to earn postsecondary credit. 
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Exhibit 4.24. 


Number of states according to various curriculum requirements for POS, by education level 


Not Don’t No 

Education level and curriculum requirement Required required know response 
Secondary 

Patt of an articulation agreement with a secondary/postsecondary partner 28 21 0 2 

Covered by a statewide articulation agreement 7 42 0 2 

Offers secondary CTE courses for academic credit toward high school diploma 12 36 0 3 

Offers secondary CTE courses with dual or concurrent enrollment 17 32 0 2 
Postsecondary 

Patt of an articulation agreement with a secondary/postsecondaty partner 25 20 1 2 

Covered by a statewide articulation agreement 12 33 1 2 

Offers postsecondary credit to secondary students through dual or concurrent 

enrollment 21 23 1 3 


Exhibit reads: Twenty-eight secondary directors reported that state-approved POS must be part of an articulation agreement with a 
postsecondary partner, while almost as many (25) postsecondary directors reported that state-approved POS must be part of an 
articulation agreement with a secondary partner. Only 7 secondaty and 12 postsecondary state directors reported that state-approved POS 
must be part of a statewide articulation agreement. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


More than half of LEAs (58 percent) offered dual enrollment or college credit opportunities 
to secondary students in at least one of their top five POS. About half (53 percent) of sec- 
ondary local directors reported that at least one of their top five POS had an articulation 
agreement with at least one postsecondary partner, and 36 percent had articulation agree- 
ments with two or more postsecondary partners (Exhibit 4.25). Between 16 and 33 percent 
of LEA directors did not know if their POS incorporated any of these features. 
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Exhibit 4.25. 
Percentage of LEAs according to inclusion of various curriculum characteristics 
in top five POS 


At least one of 


Curriculum characteristic None top five POS — Don’t know 
Part of an articulation agreement with only one postsecondary institution 25 53 22 
Part of an articulation agreement with two or more postsecondary institutions 39 36 25 
Covered by statewide articulation agreement 39 29 33 
Offers secondary CTE courses for academic credit toward high school diploma 13 71 16 


Offers secondary CTE courses for postsecondary credit (dual or concurrent at 


secondary and postsecondary levels) 23 58 19 


Exhibit reads: About 53 percent of LEA directors reported that at least one of their top five POS was part of an articulation agreement 
with only one postsecondary partner. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA Sutvey, 2009. 


The majority (76 percent) of IHEs offered postsecondary credit to secondary students 
through dual or concurrent enrollment in at least one of their top five POS. Twenty-five 
percent of IHEs had an articulation agreement with at least one school district for at least 
one of their top five POS, and 58 percent had articulation agreements with two or more 
school districts (Exhibit 4.26). Between 12 and 19 percent of IHE directors said they did not 


know if their institutions offered these options. 


Exhibit 4.26. 


Percentage of IHEs according to inclusion of various curriculum characteristics 
in top five POS 


At least one of Don’t 
Curriculum characteristic None top five POS know 
Part of an articulation agreement with only one secondary district 56 25 19 
Part of an articulation agreement with two or more secondary districts 24 58 18 
Covered by statewide articulation agreement 45 36 19 
Offers postsecondary credit to secondary students through dual or concurrent credit 12 76 12 


Exhibit reads: About 25 percent of IHE directos reported that at least one of their top five POS was part of an articulation agreement 
with one secondary partner. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: THE Survey, 2009. 


Case study interviews suggested that schools sometimes had a hard time convincing students 
and parents of the benefits of dual enrollment. According to a district administrator, “The 
biggest problem that we’ve had with dual enrollment is making folks really believe they’re 
getting the credit.” This confusion may be related to the complexities associated with the 


awatd of postsecondary credit under the old Tech Prep model. According to local adminis- 
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trators, many of the articulation agreements established under Tech Prep did not offer a 
simple and straightforward process for transcribing postsecondary credits earned at the sec- 
ondary level. As an example, one local postsecondary administrator described the cumber- 


some transcription process under a since-abandoned articulation agreement: 


The old model was students would complete a [college-] approved course at 
the [area CTE center], and as long as [they demonstrated] by the time they 
graduated that they met college level placement scores, then they could ap- 
ply for the credit and it was retroactive. They had a couple-year window of 


time. And it was confusing and not effective. It didn’t make sense. 


One local site is using a new model in which the community college approves courses for 
dual enrollment credit, and students who take the courses receive a community college tran- 
script, eliminating unnecessary waiting and paperwork. However, students are required to 


pass the college placement exam before enrolling in dual credit courses. 


In some sites, respondents attributed their lack of progress in effectively connecting second- 
ary and postsecondary systems to an inability to overcome mismatches in organizational 
structures and the logistics of scheduling and transportation. Secondary and postsecondary 
educators also had questions about who handles articulation, how to design memoranda of 
understanding (MOUs), and how best to schedule meetings. Indeed, the expert panel found 
that the most useful state guidance on articulation agreements and credit transfer included 
templates of articulation agreements and MOUs to guide local institutions through the pro- 


cess. 


Core element four: Leads to postsecondary credential or degree 
The fourth core element of Perkins IV stipulates that all POS should lead to an industry- 
recognized credential or degree at the postsecondary level to prepare students for the labor 
market and further education. However, many local directors reported that their POS did 


not lead to such outcomes or that they did not know if these options existed. 


Less than half of secondary local directors reported that at least one of their top five POS 
led to an industry-recognized or sponsored credential (47 percent), a postsecondary creden- 
tial or certificate (43 percent), or an associate’s degree (33 percent) (Exhibit 4.27). One-third 
to about one-half of secondary local directors reported that none of their POS led to such 


outcomes and between 19 and 25 percent did not know. 
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Exhibit 4.27. 
Percentage of LEA directors reporting whether top five POS led to various credentials 


At least one of 


Credential None top five POS Don’t know 
Industry-recognized or sponsored credential 35 47 19 
Postsecondary certificate 35 43 22 
Associate’s degree 46 33 22 
Bachelot’s degree oe] 22 25 


Exhibit reads: About 47 percent of LEA directors reported that at least one of their top five POS led to an industry-recognized or 
sponsored credential. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA Survey, 2009. 


Conversely, about three-quarters of postsecondary local directors reported that their POS 
led to a certificate, credential, or associate’s degree, compared to roughly 11 percent who did 
not know (Exhibit 4.28). This result might be expected, given that postsecondary programs 
ate typically designed to lead to some type of credential. Unlike secondary CTE programs 
that may be designed to expose students to a range of career options, most postsecondary 
CTE programs strive to equip students with the skills that will allow them to enter the work- 


force immediately following graduation. 


Exhibit 4.28. 
Percentage of IHE directors reporting whether top five POS led to various credentials 


At least one of 


Credential None top five POS Don’t know 
Industry-recognized or sponsored credential 10 78 12 
Postsecondary certificate 9 80 11 
Associate’s degree 14 77 9 
Bachelot’s degree 51 32 16 


Exhibit reads: The majority of IHE directors (77 to 80 percent) reported that at least one of their top five POS led to an industry- 
recognized or sponsored credential or to a postsecondary certificate or associate’s degree. About 32 percent reported that at least one of 
their top five POS led to a bachelor’s degree. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: THE Survey, 2009. 


The majority of state directors (33 secondary, 31 postsecondary) reported that all state- 
approved POS were required to be responsive to a local high-skill, high-wage, and high- 
demand field of employment (Exhibit 4.29). At the local level, slightly more postsecondary 
(86 percent) than secondary (74 percent) CTE directors reported that at least one of their 
top five POS focused on high-skill, high-demand, and high-wage jobs (Exhibit 4.30). This 
finding likely reflects the differing purposes of a secondary and postsecondary education. 
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Exhibit 4.29. 
Number of states that required all state-approved POS to be responsive to local 
high-skill, high-wage, and high-demand job areas and to offer career guidance, 


by education level 


Not Don’t No 

Education level and requirement Required required know response 
Secondary 

Responds to local high-skill, high-wage, and high-demand job atea 33 16 0 2 

Career guidance must be available 38 11 1 1 
Postsecondary 

Responds to local high-skill, high-wage, and high-demand job area 31 13 2 2 

Career guidance must be available 20 20 5 3 


Exhibit reads: At the state level, 33 secondary and 31 postsecondary state directors reported that their state required state-approved POS 
to respond to local high-skill, high-wage, and high-demand job areas. The majority of secondary (38) but less than half of postsecondary 
(20) state directors reported that their state required career guidance to be available in all state-approved POS. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Exhibit 4.30. 
Percentage of local directors reporting the inclusion of various features in top five POS, 
by agency 
At least one of 
Agency and feature None top five POS Don’t know 
LEAs 
Responds to high-skill, high-wage, and high-demand 
job areas identified by state or local agency 7 74 19 
Career guidance must be, or is, available 4 84 13 
All secondary CTE students required to select a POS 32 43 25 
IHEs 
Responds to high-skill, high-wage, and high-demand 
job areas identified by state or local agency 4 86 10 
Career guidance must be, or is, available 3 88 10 
All secondary CTE students required to select a POS n/a n/a n/a 


Exhibit reads: Among local directors, 74 percent of LEA and 86 percent of IHE directors reported that at least one of their top five POS 
responds to high-skill, high-wage, and high-demand occupational areas identified by the state or their local agency. 

n/a = Not available. 

NOTE: Estimates were weighted to reflect population estimates. 

SOURCE: LEA and IHE Surveys, 2009. 
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Supporting Components 


The Department’s POS Design Framework identifies 10 components that contribute to the de- 
velopment of effective POS. Four of these components—course sequences, credit transfer 
agreements, partnerships, and college and career readiness standards—further clarify the 
four core elements of Perkins IV discussed above. The other components, as described be- 
low, might be considered supporting components that facilitate POS development and de- 
livery. The Department identified the supporting components based on input from state 
directors and experts in POS design and implementation. The following section describes 
how staffs at the state and local levels have used the supporting components to advance 


POS development and delivery. 


Guidance counseling and advisement 
POS are intended to help students find their way through the education and career prepara- 
tion system. At the state level, 38 secondary and 20 postsecondary state directors reported 
that career guidance was required in all state-approved POS (Exhibit 4.29). At the local level, 
the majority of LEAs and IHEs provided career guidance in at least one of their top five 
POS (Exhibit 4.30). Nevertheless, some state staff interviewed for the case studies suggested 
that there was a lack of knowledge among school counselors and a lack of career materials to 
use in educating students and the public about POS. As one state director explained, “The 
challenge is making sure that a school-level person understands and uses developed POS as a 
tool to assist that local guidance counselor or career counselor. Use this as a tool, an asset. 
This is not additional work.” Another state’s CTE director described why working with 
school counselots is so vital: “We understand they’re gatekeepers and if they didn’t under- 
stand what we were doing and why we were trying to make this curricular shift, we were 


dead in the water.” 


Furthermore, the expert panel rated state-developed POS student guidance and advisement 
materials to be among the most useful of the tools recommended by the Framework. How- 
ever, panelists noted that state websites often contained information on high school course 
sequences and college entry requirements, but little to no information on career preparation 


ot POS specifically. 


Teaching and learning strategies 
POS require increased coordination between secondary and postsecondary partners and be- 
tween CTE and academic instructors. As a result, new teaching and learning strategies are 
emerging. According to one CTE instructor interviewed for the case studies, the transition 
to POS led to changes in instruction “by necessity.” A community college administrator de- 
scribed how intensive, curriculum-driven meetings between high school and community col- 
lege instructors led some instructors to the following revelation: “When we started talking 


about what we’re trying to do and... that we’re having to teach to the lower level because 
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those students aren’t coming out with [the skills needed] . . . the light bulbs went on. And 
they said, ‘Gosh, if we could give our students in the high school and in the [area CTE cen- 


ter] this, then look what we could do with the students at the college level’.” 


Some rural district staff reported turning to innovative strategies to offer their students dual 


enrollment opportunities. One local postsecondary administrator explained: 


A lot of our courses are done through distance learning and also through online 
courses. So those students can take our online courses in logistics and engineering 
processing technology, for example, through dual enrollment. And [one district] has 
a wonderful model where they have [a] paraprofessional that is in the computer lab 
with the students, and they’re working on their dual enrollment coursework during 


school time. 


One local district official described how incorporating technical and academic instruction 


was making a difference for students and leading to higher graduation rates: 


When I presented our graduation rates to our school board you could see the heads 
pop up. And one of the members asked me, ‘Why is that sor’ I responded, “Well, if I 
were to bring in a VCR and hand you a manual and have you read it and then give 
you a written test on it, have I proven that you can program that VCR?’ Until you 
can apply knowledge, you haven’t mastered anything. And it’s the same thing in 


math class. 


Technical skill assessments 
Perkins IV explicitly calls for the measurement of secondary and postsecondary CTE student 
attainment of career and technical skill proficiencies that are aligned with industry- 
recognized standards, if available and appropriate. As with standards and curriculum, states 
varied widely in their use of technical skill assessments in POS, which reflected the diversity 


of POS from one LEA (or IHE) to another; in general, there was little consistency. 


For POS, both secondary (28) and postsecondary (21) state directors reported using state- 
developed exams aligned with industry standards more than any other type of technical skill 
assessment (Exhibit 4.31). States used a variety of approaches for technical skill assessments, 
including program completion (26 secondary, 15 postsecondary) and national (19 secondary, 
17 postsecondary) or state (20 secondary, 18 postsecondary) licensing or credentialing ex- 
ams. In addition, 20 states required state-developed exams that were aligned with state tech- 


nical standards for secondary, but only four required the same for postsecondary. 
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Exhibit 4.31. 
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Number of states using various measures of technical skill attainment for POS, 


Measure of technical 
skill attainment 


State-developed exams 
Aligned with state technical standards 
Aligned with Career Cluster standards 
Aligned with industry standards 
Aligned with other standards 


Locally developed exams 
Aligned with state standards 
Aligned with Career Cluster standards 
Aligned with industry standards 
Aligned with other standards 
Industry-developed exams 
National licensing or credentialing exams 
State licensing or credentialing exams 
GPA instead of exam 


Coutse or program completion 


Other 


by education level 


Secondary 


Not 


Don’t 


Required required know 


16 


19 


20 


26 
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29 
35 
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18 
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Requited required know 
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30 
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Exhibit reads: States most often required local grantees to assess technical skill attainment via state-developed exams aligned with industry 


standards (28 secondary, 21 postsecondary). More than half of states required LEAs to use course or program completion (26 states). 


NOTE: N = 51 secondary, 48 postsecondary. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Secondary local directors also reported using a variety of instruments to assess technical 
skills, but there was little overlap between their responses and those of secondary state direc- 
tors. Many LEAs (about 35 percent) did not use state- or industry-developed technical skill 
assessments or state or national licensing or credentialing exams to assess technical skills in 
their top five POS (Exhibit 4.32), although secondary state directors reported requiring these 
assessments. Rather, over half (60 percent) of secondary local directors reported that the 
most common method of assessing POS students’ technical skills in at least one of their top 
five POS was through locally developed exams aligned with state standards; 54 percent re- 
ported using locally developed exams aligned with Career Cluster standards. About 25 per- 


cent, on average, did not know what assessments they used in their top five POS. 


Exhibit 4.32. 
Percentage of LEAs that used various measures of technical skill attainment in their 
top five POS 
At least one of 

Measure of technical skill attainment None top five POS Don’t know 
Locally developed exams aligned with state standards 19 60 21 
Locally developed exams aligned with Career Cluster standards 16 54 30 
Locally developed exams aligned with industry standards 16 53 i 
State-developed exams aligned with technical standards 36 45 19 
Industry-developed exams 36 39 25 
National licensing or credentialing exams 40 33 27 
State licensing or credentialing exams 40 33 27 


Exhibit reads: LEAs most often used locally developed exams aligned with state standards to assess students’ technical skill attainment in 
at least one of their top five POS (60 percent); LEAs used state licensing or credentialing exams least frequently (33 percent). 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA Survey, 2009. 


Survey findings revealed that approaches to technical skill assessment were even less con- 
sistent at the postsecondary level. In contrast to their state-level counterparts, IHEs most of- 
ten assessed technical skill attainment via locally developed exams aligned with industry 
standards in at least one of their top five POS (Exhibit 4.33). 
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Exhibit 4.33. 
Percentage of IHEs that used various measures of technical skill attainment in their 
top five POS 
At least one of 

Measure of technical skill attainment None top five POS Don’t know 
Locally developed exams aligned with industry standards 14 70 6 
National licensing or credentialing exams 27 58 15 
State licensing or credentialing exams 28 57 15 
Industry-developed exams 31 51 8 
Locally developed exams aligned with state standards 34 46 9 
Locally developed exams aligned with Career Cluster standards 27 42 yl 
State-developed exams aligned with technical standards 44 41 5 


Exhibit reads: IHEs most often used locally developed exams aligned with industry standards to assess students’ technical skill attainment 
in at least one of their top five POS (70 percent). 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: THE Survey, 2009. 


Variability in approaches to technical skill assessments limits the portability of technical skill 
credentials within and across states. It also impedes states’ ability to maintain useful data sys- 
tems because the technical skill test scores may not be standardized, comparable, or accurate. 
The expert panel found states’ guidance on this component of POS to be the least useful 
and most in need of improvement. The few states whose guidance on technical skill assess- 
ments the panel found useful provided lists of available assessments by career area, source, 
and costs, as well as links to state licensing agencies and industry websites with certification 


requirements. 


As with the survey findings, the case studies surfaced a range of opinions on technical skill 
assessments. One secondary state administrator established incentives for local districts to 
offer industry certifications on the grounds that these would help students be more market- 
able. A counterpart in another state allowed local subgrantees to provide industry certifica- 
tions but would not require them to do so because “there are some areas that simply don’t 


have industry certification or licenses.” 


In several case study states, secondary educators at both the state and local levels questioned 
the practice of administering assessments to provide industry certifications to secondary 
POS participants. The CTE director in one case study state argued that grade point average 
was the best indicator of technical skill attainment: “We measure technical skill proficiencies 
through multiple demonstrations .. . throughout the semester. Why would I take one test 


and say that your one paper-and-pencil test is your level of proficiency?” 


The state director in another state convened teams of CTE instructors and industry experts 


to develop common statewide capstone assessments for POS, starting with high-enrollment 
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cluster areas. He explained that the state agency had “given up” on using commercially avail- 
able exams to demonstrate skill attainment because of the difficulty in obtaining data from 
the testing entity. He also noted that the state-developed assessments were intended to drive 
curriculum change as POS continued to develop. However, he also acknowledged resistance 
to common assessments from some directors of regional CTE centers because they are not 


required specifically by the Perkins IV legislation. 


The state director from another case study state said it would be valuable for the Depart- 
ment to provide more leadership on assessments. “We got some push-back [from local 
agencies] . .. because the legislation simply says you'll provide a measurement of students’ 
skill attainment, but it doesn’t mandate formal assessments. We couldn’t say, “This is re- 
quired of Perkins.’. .. If someone savvy enough challenged that in a tight budget, we’d have 
to back off.” Another secondary state official said, “We have states that have spent millions 
of dollars on assessment. They don’t even know if it’s going to be accepted, because the feds 


haven’t identified in enough detail what will be accepted.” 


Accountability and evaluation 
Although Perkins IV does not require detailed reporting on POS outcomes, states must en- 
sure that local suberantees have at least one POS that complies with legislative requirements. 
In nearly all states (47 secondary, 41 postsecondary) state agency staff were responsible for 
monitoring the implementation of state-approved POS for Perkins IV compliance; local staff 


share some of these responsibilities (Exhibit 4.34). 


Exhibit 4.34. 
Number of states according to various individuals responsible for monitoring local POS 
implementation for Perkins [V compliance, by education level 


Individuals responsible Secondary Postsecondary 
State agency staff 47 41 
LEA administrators of CTE 29 16 
Secondary teachers or administrators 16 8 
Postsecondary faculty or administrators 11 20 
Advisory committees 11 6 
Other agency or individual 3 3 
Not applicable or not done 1 3 
No tesponse 1 1 


Exhibit reads: State agency staff were most often responsible for monitoring local POS implementation (47 secondary, 41 postsecondary). 
LEA administrators also were responsible for monitoring POS implementation within the secondary education level in 29 states. 
Postsecondary faculty or administrators were responsible for monitoring POS implementation in the postsecondary sector in 20 states. 
NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Case study visits revealed that three of the six states did not yet have an evaluation process 
for measuring POS quality; state officials merely documented the presence or absence of 
POS by reviewing local Perkins IV plans for evidence of whether they addressed the four leg- 
islated elements. Staff interviewed from the other three states concluded that their POS were 
of good quality, based on the monitoring systems currently in operation, and emphasized the 
effort and attention that state staff had given to technical assistance and standard setting 
around POS. Administrators in several case study states said they had incorporated POS re- 
view into consolidated program audits that included other non-CTE programs at the sec- 


ondary and postsecondary levels. 


Survey findings indicated that the majority of states monitored and evaluated POS through 
state reporting requirements or state agency staff visits to individual sites (Exhibit 4.35). Of 
those states that conducted state monitoring activities, most used both staff visits and state 
reporting requirements at both the secondary (33) and postsecondary (21) education levels 
(Exhibit 4.36). 


Exhibit 4.35. 
Number of states according to various POS monitoring and evaluation activities, 


by education level 


Secondary Postsecondaty 

Monitoring and evaluation activity Monitoring Evaluation Monitoring Evaluation 
State agency staff visit individual sites 39 34 30 24 
LEA administrators visit individual sites" 18 14 n/a n/a 
Secondary district office staff visit individual sites n/a n/a 5 4 
Through state reporting requirements for POS 42 39 28 28 
Other approach 5 10 8 11 

Not applicable or not done 2 4 8 - 

No response 1 1 1 1 
Exhibit reads: States most often used staff visits and state reporting requirements to monitor and evaluate local POS activities. 


n/a Not applicable. 
NOTE: N = 51 secondary, 48 postsecondary. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Exhibit 4.36. 
Number of states according to use of staff visits and state reporting requirements to monitor 


and evaluate POS, by activity and education level 


Monitoring 


State agency State reporting 
staff visits requirements 
Secondary 
6 o) 
State agency State reporting 
staff visits requirements 
Postsecondary 
9 7 
Evaluation 
State agency State reporting 
staff visits requirements 
Secondary 
6 11 
State agency State reporting 
staff visits requirements 
Postsecondary 
5 9 


Exhibit reads: Most state directors reported using both state agency staff visits and reporting requirements to monitor and evaluate 
POS. 
SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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Legislation and policies 
While all state officials outlined procedures for POS development in their state plans, most 
states did not have legislation to support POS for secondary (33) or postsecondary (34) edu- 
cation (Exhibit 4.37). Some states had made changes to administrative policies (20 second- 
ary, 20 postsecondary) to support POS development, and some had not made any policy 
changes by the time of the survey (12 secondary, 20 postsecondary). 


Exhibit 4.37. 
Number of states that adopted legislation and modified or expanded policies supporting 
POS since passage of Perkins IV, by education level 


Yes, for reasons No, already 
Yes, in response other than have 
to Perkins Perkins supporting 
Education level reauthorization reauthorization legislation No Don’t know 
Secondary 
State legislation 2 3 12 33 1 
Policy modification or expansion 20 12 6 12 1 
Postsecondary 
State legislation 5 1 6 34 2 
Policy modification or expansion 20 5 3 20 0 


Exhibit reads: Only a few states adopted legislation to support POS since the passage of Perkins IV (five secondary, six postsecondary). In 
contrast, more than half (32 secondary, 25 postsecondary) modified or expanded policies in support of POS. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


Case study findings suggested that state legislation supporting POS may facilitate POS de- 
velopment. A state director from one state with such legislation reported that CTE is seen 
by the state officials as “very important in the educational process .. . [there are] ways that 
[CTE is] embedded in our legislature, in our laws, in our department policies, and where we 
stand today.” State officials in this state cited plans to offer all CTE through POS by 2012. 


Some state administrators reported meeting resistance from some local subgrantees in initia- 
tives not explicitly required by Perkins IV. One state director acknowledged being burdened 
by legislative requirements that were not accompanied by additional funding: “These people 
[local subgrantees] have no problem picking up the phone and calling [the Department] and 
saying, “This is another unfunded mandate. I don’t have to do this’.” 
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In “local control states’”—1.e., those in which LEAs and IHEs have primary decision-making 
authority for educational programs—state agencies cannot mandate curriculum or the par- 
ticular form that POS will take. In these circumstances, state agency staff described attempts 
to provide policy guidance and leadership in the local POS development and implementation 
process. As one state director acknowledged, local control “doesn’t excuse an absence of 
state leadership.” Even if local control states cannot mandate a process for developing POS, 
state leaders can “shine a light on what is ideal’ and provide “encouragement to go in the 


right direction.” 


Professional development 
Because POS are new in Perkins IV and require more investment at all levels, states had to 
provide support to local agencies in developing and implementing POS. The majority (47 
secondary, 35 postsecondary) of states provided training and professional development to 
local staff as their primary type of support (Exhibit 4.38). In most states, state staff at both 
the secondary and postsecondary levels created POS models or templates (44 secondary, 31 
postsecondary). A majority of states developed guides for aligning CTE content with aca- 
demic and technical standards (35 states) and curriculum guides or other materials (32 states) 


for secondary education. 


Exhibit 4.38. 
Number of states according to various actions taken to align local POS development with 
Perkins IV core elements, by education level 


Action Secondary Postsecondary 
Providing training and professional development to LEA/IHE staff 47 35 
Creating POS models or templates 44 31 
Securing secondary and postsecondary agreement on specific POS elements 38 32 
Designing guides for aligning CTE content with academic standards 35 16 
Designing guides for aligning CTE content with technical standards 35 16 
Requiring LEAs/THEs to provide evidence of alignment in local plans 33 25 
Consulting Career Clusters developed by SCCI 33 26 
Developing curriculum guides or other materials 32 10 
Other 4 n/a 
No response 1 2 


Exhibit reads: States most often provided training and professional development to local grantees to ensure POS aligned with the 
Perkins IV core elements (47 secondary, 35 postsecondary). 

n/a = Not available. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 
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More than half of local directors reported that their state offered at least one type of profes- 
sional development for instructors, administrators, and counselors, and almost as many te- 
ported that these groups participated in such activities. Specifically, 73 percent of secondary 
local directors reported that their state offered training to administrators and within those 
states, 79 percent of LEAs reported that local administrators participated in training (Exhibit 
4.39). Similarly, 82 percent of IHE directors indicated that the state offered training to post- 
secondary administrators, and 86 percent of IHEs in states that offered training said that 


their administrators participated in training. 


Exhibit 4.39. 
Percentage of local directors reporting whether their state offered and local staff participated 
in at least one type of professional development focused on POS, by agency and type of staff 


Among those reporting professional 


At least one type of development offered, percentage 
professional development offered that had staff participate 
Percent Don’t Percent Don’t 
Agency and type of staff responding Yes know responding Yes know 
LEAs 
Teachers and faculty 81 8 85 6 
Administrators 73 12 79 9 
Counselors and advisors 68 16 76 11 
THEs 
Teachers and faculty 77 10 86 6 
Administrators 82 8 86 8 
Counselors and advisors 53 27 79 12 


Exhibit reads: About 81 percent of LEA directors reported that their state offered professional development focused on POS for 
teachers. 

NOTE: Estimates were weighted to reflect population values. Professional development includes the following types of activities: 
state/regional conferences, local workshops, online webinars, one-on-one support, and other types of professional development. 
SOURCE: LEA and IHE Surveys, 2009. 
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State directors at both levels reported that local administrators and CTE instructors were the 
most likely audiences for professional development focused on POS (Exhibit 4.40). Postsec- 
ondary state directors reported that postsecondary counselors were the least likely to receive 


such training from their state. 


Exhibit 4.40. 
Number of states that offered professional development focused on POS, by education level 
and type of staff 


At least one type of 


professional 
development 
Education level and type of staff offered None Don’t know No response 
Secondary 
Administrators 48 0 1 2 
Secondary CTE teachers 48 0 1 2 
Secondary counselors 45 0 3 3 
Secondary academic teachers 41 3 i 2 
Postsecondary 
Administrators 43 2 1 2 
Faculty 40 4 2 
Counselors a7 14 4 


Exhibit reads: Almost all (48) secondary state directors reported offering at least one type of professional development on POS for local 
administrators and for secondary CTE teachers; slightly fewer reported offering professional development on POS for secondary 
counselors or for academic teachers. At the postsecondary level, 43 states reported offering professional development on local POS for 
administrators, and fewer states reported offering professional development for faculty and counselors. 

NOTE: Professional development includes the following types of activities: state and regional conferences, local workshops, online 
workshops and webinars, and one-on-one support. N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


In addition to professional development, states also provided technical assistance to local 
suberantees. States most often offered technical assistance through POS templates or guide- 
lines for both secondary (45 states) and postsecondary (32 states) (Exhibit 4.41). While states 
offered technical assistance on a variety of other topics, Career Clusters was the only other 


technical assistance topic offered frequently at both levels (36 secondary, 26 postsecondary). 
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Exhibit 4.41. 
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Number of states that introduced new technical assistance activities on various POS topics, 


by education level: 2007—08 or 2008-09 


POS topic Secondary Postsecondary 
POS templates or guidelines 45 32 
Career guidance and counseling 39 17 
CTE content standards 38 17 
Career Clusters 36 26 
Academic and CTE curriculum integration 36 22 
Technical skill assessments 34 27 
Data systems for monitoring student progress 34 24 
Secondary and postsecondary curriculum alignment 33 27 
CTE curticulum development guidelines 26 9 
Aligning standards and assessments 25 10 
Other 3 2 
No response 1 1 


Exhibit reads: States introduced a variety of new technical assistance opportunities to local grantees in support of POS development in 


2007-08 and 2008-09, although offering POS templates or guidelines was the most frequent form for both the secondary and 
postsecondary education levels. 

NOTE: N = 51 secondary, 48 postsecondary. 

SOURCE: Secondary and Postsecondary State Director Surveys, 2009. 


At the secondary level, survey findings revealed some incongruence between technical assis- 
tance opportunities offered by state officials and local participation in them. For example, 33 
states offered technical assistance to LEAs on the alignment of secondary-postsecondary 
curriculum (Exhibit 4.41), but only 57 percent of LEA directors reported receiving state- 
sponsoted technical assistance on this topic (Exhibit 4.42). Similarly, 39 states offered tech- 
nical assistance to LEAs on career guidance and counseling (Exhibit 4.41), while only 63 
percent of LEA directors reported receiving state-sponsored technical assistance on this top- 
ic (Exhibit 4.42). 


More LEA than IHE respondents reported receiving technical assistance from the state on a 
variety of topics related to POS. These topics included Career Clusters, CTE content stand- 
ards, CTE curriculum development guidelines, and academic and CTE curriculum integra- 
tion (Exhibit 4.42). Postsecondary local directors reported recetving more technical 
assistance than secondary local directors on secondary-postsecondary curriculum alignment 


and POS templates or guidelines. 
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Exhibit 4.42. 


Percentage of local directors who reported whether their state offered and local staff 


participated in technical assistance on various POS topics, by agency 


LEAs IHEs 
“State —s Staff ==“ (ss:s*«éCState:~=:*~=<“‘«‘S tA 

POS topic offered patticipated offered patticipated 
POS templates or guidelines 74 73 78 84 
Career Clusters 82 83 76 80 
CTE content standards 81 76 63 78 
CTE curticulum development guidelines 70 74 58 78 
Academic and CTE curriculum integration 65 79 53 80 
Secondary and postsecondary curriculum alignment 57 76 64 90 
Technical skills assessments 55 77 47 80 
Career guidance and counseling 63 83 44 87 
Data systems for monitoring student progress 43 80 42 79 
Aligning standards and assessments 54 69 47 78 
Other 1! 91! 3! 86! 


Exhibit reads: About 74 percent of secondary directors reported that their state offered technical assistance on POS templates, and about 


73 percent reported that thier staff participated in professional development on this topic. 
! Interpret data with caution. These items have a response rate below 85 percent. 

NOTE: Estimates were weighted to reflect population values. 

SOURCE: LEA and IHE Surveys, 2009. 


The case study findings suggested that some local staff were generally satisfied with the 
technical assistance offered by the state, but others found it insufficient or not equally valua- 
ble to every stakeholder. In one state, a postsecondary administrator summed up his view by 
saying, “I’m sure they’re [the state agency] underfunded and have a lot of responsibilities, but 


we're pretty much on out own.” 


In many case study states, travel budget cuts at the state level had curtailed the ability of state 
staff to visit local subgrantees and provide on-site assistance in developing POS. In one case 
study state, budget difficulties had reduced staff at the state department to the extent that 
only one staff member remained to provide services to local subgrantees. In referring to the 
state education department, a local director said, “They’re not a big department... and 
there’s only so much you can expect out of those few people.” Local recognition of the neg- 
ative effects of the economy on state departments was widespread but met with varying de- 


grees of acceptance. 


The expert panel reviewed many types of POS guidance provided by states, but the primary 
type reviewed was that which was available on state websites (see Appendix B). Overall, the 
panelists agreed that state POS guidance most helpful to local subgrantees would include 


easily navigable, user-friendly websites organized around CTE, POS, and technical assistance 
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and include contact information for state staff members who could provide assistance with 
specific topics. The panel found disorganized websites or websites that offered sparse, out- 
dated, and confusing information, and particularly a lack of POS-specific information to be 
least helpful. Recognizing that local program staff often rely on the Internet for information 
on Perkins IV requirements, panel members judged poor-quality websites to be a substantial 


obstacle to POS design and implementation. 


The expert panel concluded that LEAs and IHEs needed increased state guidance on many 
POS components and identified a number of areas in which additional support was particu- 
larly critical. These areas included developing secondary and postsecondary partnerships, 
improving opportunities for credit transfer, developing teaching and learning strategies to 


support POS instruction, and identifying options for technical skill assessment. 
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Chapter 5. Conclusion 


With each reauthorization, Congress has worked to refine and improve the provisions of the 
Carl D. Perkins Career and Technical Education Act (Perkins).'These modifications have, over 
time, expanded the scope of the legislation, tightened administrative and compliance expec- 
tations, and introduced new policy initiatives to improve the effectiveness and reach of ca- 
reer and technical education (CTE). Although federal funding constitutes only a fraction of 
the resources expended on CTE nationwide, Department policies and administrative guid- 
ance have some effect on the organization, administration, and delivery of CTE services at 


the state and local levels. 


Since its introduction in 1984, Perkins legislation has gradually evolved in response to chang- 
ing educational conditions and policy priorities. While Perkins [VV retains many of the key 
provisions and statutory requirements contained within preceding legislation, not all of them 
have remained static. As detailed in this report, Perkins IV introduced new legislative initia- 
tives intended to increase state flexibility in using resources, extend performance and ac- 
countability expectations to the local level, and promote the development and 
implementation of POS to improve the transitions that students make from secondary to 


postsecondary education. 


Legislative Advances 


Over time, federal efforts to expand accountability, enhance the integration of academic 
knowledge and technical skills, and improve program effectiveness through initiatives such 
as Tech Prep and POS have fostered innovation and helped shape CTE policies and practic- 
es at the state and local levels. Legislative benefits identified in this study include the follow- 


ing: 


1. State agency staff and local practitioners rely on federal Perkins IV funds to support statewide program 


improvement efforts and to enhance local program quality. 


Continuous improvement is a guiding principle of the legislation, and state directors rely on 
Perkins IV state leadership funds to support statewide technical assistance and program im- 
provement efforts. States also rely on Perkins IV state administration resources to offset 
staffing and other costs associated with legislative oversight, monitoring, and grant admin- 


istration. Local educators, in turn, depend on Perkins IV to fund the development and im- 
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plementation of new initiatives, with many reporting that they would have difficulty keeping 
instructional programs current without federal support for CTE. In particular, LEA and 
IHE staff rely on Perkins IV funds to purchase and service classroom equipment, as well as 
to keep curriculum and instruction up-to-date with changing technical requirements in their 
fields. 


2. The refinement of Perkins IV accountability systems and extension of performance expectations to local 


subgrantees have focused state and local administrators’ attention on student and program outcomes. 


Perkins IV tequites that states report valid and reliable data and outlines a refined set of sec- 
ondary and postsecondary core indicators to assess CTE student and program outcomes. 
The introduction of new Tech Prep indicators expands accountability requirements to in- 
clude programs supported by the Title H grant. Perkins IV also extends performance expec- 
tations to the local level: states must now negotiate individual performance targets with each 
LEA and IHE or require that programs adopt state-established targets, which has helped di- 
rect states and local subgrantees’ attention toward achieving agreed-upon performance lev- 
els. Local agencies falling short of negotiated targets face progressive sanctions, beginning 
with the requirement that they develop a local program improvement plan to address identi- 


fied deficiencies and culminating in the loss of some or all of their federal funding. 
3. Perkins IV promotes program alignment across the secondary and postsecondary education levels. 


Perkins IV introduces the requirement that all local subgrantees offer one or more POS to 
qualify for continued funding. This new initiative, which builds upon existing Tech Prep pol- 
icies that promote alignment across the secondary and postsecondary education levels, is 
leading state and local administrators to reconsider how CTE instruction is organized and 
delivered. Although the scope and components of POS vary considerably among states, 
statutory requirements, in conjunction with guidance contained within the Department’s 
POS Design Framework, encourage secondary educators to coordinate with postsecondary fac- 
ulty in order to align and sequence secondary and postsecondary education coursework. 
Moreover, statutory changes that allow states to merge their Title II (Tech Prep) with Title I 
(basic grant) funding have increased state flexibility, prompting some states to undertake 


statewide efforts to better alien CTE offered at the secondary and postsecondary levels. 


Reauthorization Issues 


While Congress used reauthorization to refine existing provisions and advance new legisla- 
tive opportunities, state and local administrators identified a number of issues that compli- 
cate the implementation of the legislation. Reauthorization offers Congress an opportunity 
to address identified challenges and consider new directions to achieve the legislation’s in- 


tended purposes. 
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1. Congress has introduced new requirements with each Perkins reauthorization, adding to the administra- 


tive and data-reporting responsibilities of states and local subgrantees. 


Perkins IV expands state performance accountability expectations and introduces POS as a 
new program delivery strategy. These new statutory requirements have added to the adminis- 
trative duties of state and local staff. For example, while the expectation that all providers of- 
fer one or more POS has promoted program alignment across education levels, this 
innovation has intensified state technical assistance obligations, along with the expectations 
of state and local administrators to conduct compliance monitoring. These increased admin- 
istrative tasks occur against the backdrop of an economic recession that has led state and lo- 
cal agencies to institute staffing cuts and mandatory furloughs. Moreover, while federal 
funding for CTE has increased in nominal dollars, resource allocations have declined in real 
terms, meaning that state and local administrators must take on added responsibilities in a 


challenging resource environment. 


2. Accountability data are not comparable across states and, in some instances, are difficult to collect due to 


state and local capacity and infrastructure constraints. 


Perkins IV statutory provisions and the Department’s nonregulatory guidance offer states 
flexibility in interpreting accountability requirements, contributing to variation in practices 
for collecting and reporting data across states. This effort to preserve state control prevents 
the Department from aggregating data to obtain a national perspective. State and local ad- 
ministrators also face challenges in obtaining required data for some measures. In particular, 
limitations in the capacity of state and local student information systems, concerns over pto- 
tecting students’ privacy, and lack of access to some required data continue to affect the abil- 


ity of some states to implement all Perkins accountability requirements. 


3. Perkins IV requirements for POS are broadly defined and, until recently, lacked guidance from the 


Department, consequently, POS vary considerably across states. 


Prior to the Department’s release of the POS Design Framework in January 2010, states relied 
on the four core elements of POS as defined in the legislation. Study findings indicate that 
while most state and local agencies were actively engaged in POS development and imple- 
mentation, the scope and quality of programs vary dramatically. This is due to several rea- 
sons, including lack of state legislation and policies directing local POS creation, limited 
understanding or misconceptions of POS among state and local staff, logistical challenges 
that complicate program coordination across the secondary and postsecondary levels, and a 
lack of fiscal and human resources. Moreover, while secondary and postsecondary education 
administrators have cooperated somewhat in jointly planning POS, Perkins IV does not ear- 
mark resources for cross-level coordination and does not hold educators responsible for as- 


sessing student and program outcomes. Lack of clear definitions on what constitutes a POS 
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student and the absence of standardized reporting requirements relating to POS further con- 
found state implementation efforts. To date, relatively few state or local Perkins subgrantees 
are capable of providing accurate counts of the students who participate in POS or their 


outcomes. 


In Summary 


Federal Perkins legislation continues to play an important role in promoting continuous im- 
provement in the organization and delivery of CTE services. The dedicated financial support 
that Perkins IV provides is critical to maintaining state CTE leadership and keeping local 
CTE programs and equipment up-to-date with the skill demands of a dynamic workplace. 
Accountability requirements continue to hold states and local subgrantees responsible for 
achieving results. Moreover, the introduction of POS is promoting alignment and coordina- 


tion of CTE services across the secondary and postsecondary levels. 


In many respects, Perkins has arrived at a crossroads. Successive reauthorizations have added 
to the legislation’s requirements at a time when many states are struggling to maintain CTE 
programs and staff, making it increasingly challenging for state administrators and local pro- 
gram directors to implement Perkins IV's components and monitor subgrantees effectively. 
While the Department has attempted to improve the validity and reliability of Perkins IV ac- 
countability data, consistency of information varies within and across states, as well as across 
specific indicators. This lack of consistency may be traced to the flexibility inherent in the 
legislation, the nonregulatory nature of the Department’s guidance, and continuing challeng- 
es to state and local collection of data. And while POS offer a new framework for organizing 
the content and delivery of CTE coursework, Perkins IV resource distribution formulas are 
not designed to promote this initiative, for example, by providing cross-sector incentives to 
encourage collaboration. The absence of performance accountability expectations also pre- 
vents federal and state policymakers from assessing the potential contribution that these 


programs may offer. 


As Congress begins its reauthorization deliberations, careful consideration should be di- 
rected to the scope and specificity of the current legislation. While policymakers have histor- 
ically used reauthorization to improve and redirect the legislation, for example, by adding 
expectations for POS development, the continued addition of new requirements complicates 
state and local administration and program management. Statutory flexibility also has, to 
date, allowed states to respond differently to grant requirements. Congress may wish to use 
reauthorization to assess the extent to which current legislative provisions and state flexibil- 


ity to respond to them contribute to achieving the legislation’s intended purposes. 
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Glossary 


Allocation—The amount of Perkins grant money that a state grantee or local subgrantee re- 


ceives. 


Allotment—A state grantee’s or local subgrantee’s share of Perkins funding, determined by 


grant formula. 


Atea CTE centers—Secondaty institutions that provide career and technical education 
(CTE) instruction to students who receive all or most of their academic instruction at their 
home high school. An area CTE center often serves multiple high schools within multiple 


school districts. 
Basic grant—Funds allocated to states under Title I of Perkins IV. 


Career and technical education (CTE)—“[O]rganized educational activities that—(A) of- 
fer a sequence of courses that—(1) provides individuals with coherent and rigorous content 
aligned with challenging academic standards and relevant technical knowledge and skills 
needed to prepare for further education and careers in current or emerging professions; (1i) 
provides technical skill proficiency, an industry-recognized credential, a certificate, or an as- 
sociate degree; and (tit) may include prerequisite courses (other than a remedial course) that 
meet the requirements of this subparagraph; and (B) include competency-based applied 
learning that contributes to the academic knowledge, higher-order reasoning and problem- 
solving skills, work attitudes, general employability skills, technical skills, and occupation- 
specific skills, and knowledge of all aspects of an industry, including entrepreneurship, of an 
individual” (Sec. 3(5)). 


Consortia—Local education agency (LEAs) and institutions of higher education (IHEs) 
Perkins applicants that do not meet the Title I minimum allocation criteria are required to en- 
ter consortia and work together to provide programs that are of sufficient size, scope and 
quality to be effective, unless the state waives this requirement (Sec. 131(d); Sec. 132(a),(0)). 
Title II (Tech Prep) funds are awarded to consortia of one or more secondary providers and 


one or more qualifying postsecondary institutions (Sec. 203(a)). 


Eligible agency—“[A] state board designated or created consistent with state law as the 
sole state agency responsible for the administration of career and technical education in the 
state or for the supervision of the administration of career and technical education in the 
state” (Sec. 3(12)). 
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Eligible recipient—[A] local educational agency (including a public charter school that 
operates as a local educational agency), an area career and technical education school, an ed- 
ucational service agency, or a consortium, eligible to receive assistance under section 131; or 
an eligible institution or consortium of eligible institutions eligible to receive assistance under 
section 132” (Sec. 3(14)). 


Fiscal year—Begins on October 1 of one calendar year and ends on September 30 of the 
next year and is denoted by the second calendar year. For example, FY 2008 refers to the 12- 
month period between October 1, 2007, and September 30, 2008. 


Institution of higher education (IHE)—According to Sec. 3(18) of Perkins IV, this term is de- 
fined in Sec. 101 of the Higher Education Act of 1965 as follows: “[T]he term ‘institution of 
higher education’ means an educational institution in any State that—(1) admits as regular 
students only persons having a certificate of graduation from a school providing secondary 
education, or the recognized equivalent of such a certificate; (2) is legally authorized within 
such State to provide a program of education beyond secondary education; (3) provides an 
educational program for which the institution awards a bachelor’s degree or provides not 
less than a 2-year program that is acceptable for full credit toward such a degree; (4) is a pub- 
lic or other nonprofit institution; and (5) is accredited by a nationally recognized accrediting 
agency of association, or if not so accredited, is an institution that has been granted preac- 
creditation status by such an agency or association that has been recognized by the Secretary 
for the granting of preaccreditation status, and the Secretary has determined that there is sat- 
isfactory assurance that the institution will meet the accreditation standards of such an agen- 


cy or association within a reasonable time” (Sec. 101(a)). 


Local director—Refers to the individuals at the secondary and postsecondary education 


levels who responded to the LEA and IHE surveys conducted for this study. 


Local education agency—According to Sec. 3(19) of Perkins IV, this term is defined in Sec. 
9101 of the Ementary and Secondary Education Act of 1965 as follows: “[T]he term ‘local educa- 
tional agency’ means a public board of education or other public authority legally constituted 
within a State for either administrative control or direction of, or to perform a service func- 
tion for, public elementary schools or secondary schools in a city, county, township, school 
district, or other political subdivision of a State, or of or for a combination of school districts 
ot counties that is recognized in a State as an administrative agency for its public elementary 
schools or secondary schools” (Sec. 9101(26)). 


Nontraditional—‘“[O]ccupations or fields of work, including careers in computer science, 
technology, and other current and emerging high skill occupations, for which individuals 
from one gender comprise less than 25 percent of the individuals employed in each such oc- 
cupation or field of work” (Sec. 3(20)). 


GLOSSARY 


Outlying area—‘[T]he United States Virgin Islands, Guam, American Samoa, the Com- 
monwealth of the Northern Mariana Islands, and the Republic of Palaw” (Sec. 3(21)). 


Program year—Begins on July 1 of one calendar year and ends on June 30 of the next year 
and is denoted by the starting and ending years. For example, program year 2007—08 refers 
to the 12-month period between July 1, 2007, and June 30, 2008. 


Programs of study—“[I|ncorporate secondary education and postsecondary education el- 
ements; include coherent and rigorous content aligned with challenging academic standards 
and relevant career and technical content in a coordinated, nonduplicative progression of 
courses that align secondary education with postsecondary education to adequately prepare 
students to succeed in postsecondary education; may include the opportunity for secondary 
education students to participate in dual or concurrent enrollment programs or other ways to 
acquire postsecondary education credits; and lead to an industry-recognized credential or 


certificate at the postsecondary level, or an associate or baccalaureate degree” (Sec. 122(1) (a). 


Reserve fund—States may reserve not more than 10 percent of the 85 percent of Title I 
funds to distribute to local programs in rural areas, and areas with high numbers or percent- 
ages of CTE students (Sec. 112(a)(1),(©). 


Reporting year—The program year ending on June 30 prior to the December 31 Consoli- 
dated Annual Report (CAR) submission deadline. For the December 2010 CAR submission, 
the reporting year is 2009-10 (July 1, 2009, through June 30, 2010). 


State—Perkins IV defines a “state” as “each of the several States of the United States, the 
District of Columbia, the Commonwealth of Puerto Rico, and each outlying area” (Sec. 
3(30)). The term “state” as it is used in this report includes the 50 states of the United States; 
the District of Columbia; the Commonwealth of Puerto Rico; and the outlying areas of the 
USS. Virgin Islands, Guam, American Samoa, the Commonwealth of the Northern Mariana 
Islands, and the Republic of Palau. 


State director—Refers to the individuals at the secondary and postsecondary education lev- 
els who responded to the secondary and postsecondary state director surveys conducted for 


this study. 


Subgrantee—An eligible recipient or eligible institution that receives a Perkins IV grant 


from the state eligible agency. 


Tech prep—Refers to the content of Tech Prep programs described in Sec. 3(32) of Perkins 
IV. 
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Appendix A. Methodology 


Obtaining the array of quantitative and qualitative data needed to address the study topics 
required the use of a mixed-methods study design. Over the course of this 36-month study, 
the research team collected original data from state and local career and technical education 
(CTE) directors, administrators, faculty, and staff, and compiled extant data maintained in 
U.S. Department of Education (Department) databases and available online. The variety of 
approaches enabled the research team to collect different types of information, at varying 
levels of specificity, from individuals with diverse roles and responsibilities in implementing 
the provisions of the Car/ D. Perkins Career and Technical Education Act of 2006 (Perkins IV). 


This appendix summarizes the research strategies used for the study. 


Original Data Sources 


The research team relied on existing research whenever possible; however, to meet the spe- 
cific requirements of the study, the team also collected original data through a series of sur- 
veys and site visits (Exhibit A.1). A nationwide survey of state and local CTE directors 
provided information on the design and implementation of Perkins IV fiscal, accountability, 
and programs of study (POS) systems.! Surveys of local directors at a representative sample 
of local education agencies (LEAs) and institutions of higher education (IHEs) offered in- 
sights into local perspectives and systems. In addition to the state director surveys, the study 
team asked each state to supply data on their Perkins IV Title I and Tech Prep grant alloca- 
tions to local programs. These data enabled the research team to quantify the number, size, 
and distribution of grants, controlling for provider characteristics and consortia participa- 


tion. 


Concurrent with survey administration, the research team conducted case study visits to six 
representative states. The case studies were designed to elicit more detailed insights into how 
Perkins IV legislative language was translated into state administrative policies and regula- 
tions, and its affect on program operations and staffing. The six states were selected for their 
geographical distribution and characteristics of POS implementation, including their per- 


ceived quality and scope of system adoption and their method of designing programs (cen- 


! The term “state director” as it is used in this report refers to the individuals at the secondary and 
postsecondary education levels who responded to the secondary and postsecondary state director sur- 
veys. The term “local director” refers to the individuals at LEAs, IHEs, and area CTE centers who re- 
sponded to the LEA and THE surveys. 
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tralized versus decentralized. Study teams visited the eligible agency and three local POS 
partnerships within each state, selected to be representative of an urban, suburban and rural 
community. Local visits included interviews with representatives of the secondary LEA, af- 
filiated IHE, and any business, industry, or labor group collaborating to offer a state- 
approved POS. 


The requirement that all local subgrantees offer one or more POS is a significant legislative 
innovation in Perkins IV. To assess the quality of state implementation support to local sub- 
grantees, the researchers convened an expert panel to review state POS guidance and materi- 
als gathered from the state director surveys and website searches. During a series of 
facilitated teleconference calls, expert panelists shared their ratings of state guidance and ma- 
terials and offered suggestions for improving communications relating to POS design and 


implementation. For more information about the expert panel’s findings, see Appendix B. 


Exhibit A.1. 
Data sources 


Approach Content area Source 


Account- 


A-2 


Original data sources Fiscal ability POS 


Surveys 


XxX 


Xx 


Xx 


State directors: Secondary and postsecondary 


Local directors: Secondary and postsecondary 


Financial allocation data XxX State directors: Secondary and postsecondary 
Case studies (six states) Xx Xx XxX Eligible agency 
State directors: Secondary and postsecondary 
State administrative staff 
Local partnerships (three agencies in each state) 
Local directors: LEA and [HE 
Teachers and faculty: LEA and [HE 
Business, industry, and labor partners 
Expert panel XxX Websites 
State guidance and materials 
Extant data sources 
Department administrative x www.perkinsinfo.com 
databases www.perkinscar.com 
Department soutces Xx Nontregulatory guidance 
Annual Perkins reports to Congress 
Findings from Department-sponsored 
technical assistance to states 
Existing research and Xx Xx Xx State websites 


online information 


Education literature 


Exhibit reads: The research team relied upon a variety of original and extant data to address the three content areas of the study. 
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State and Local Surveys 


The study team administered online surveys to the universe of state directors and a repre- 
sentative sample of local directors to gather information on the implementation of Perkins 
IV fiscal, accountability, and POS provisions.*? Survey data allowed researchers to under- 
stand how the Department communicated Perkins IV legislative requirements to states and 
how states interpreted and implemented legislative requirements. Information also was col- 
lected on state and local administrative policies, technical assistance support, and student 


participation and program performance. 


It should be noted that survey participants were asked to report on data from the 2008-09 
program year. Although this was the second year of Perkins IV, it was the first year following 
states’ approval of their five-year plan for offering CTE services. Data collection, therefore, 
focused on states’ initial efforts to implement Perkins IV. Further, the survey of state direc- 
tors represents nearly every state, but the survey of local directors used a random, nationally 
representative sample of LEAs and IHEs. As a result, local director data offer insight into 


national perspectives, but may not be descriptive of conditions within a given state. 


Survey planning and development 
The study team first met with Department representatives to obtain copies of survey instru- 
ments and data collection methodologies used in the spring 1992 and summer 2001 National 
Assessment of Vocational Education (NAVE) surveys. The researchers reviewed these doc- 
uments to identify questions that could be used to replicate trend analyses contained in earli- 
er national assessments. With few exceptions, these questions were adopted verbatim or with 


minor textual edits into the 2009 survey instrument. 


In November 2008, the researchers met with National Center for Education Statistics 
(NCES) staff after determining that NCES was planning to survey secondary and postsec- 
ondary state directors in early 2009. Because this effort would have overlapped with the tim- 
ing and content of this study’s survey, NCES staff agreed to delay their survey to reduce the 
state reporting burden and refocus their study effort on a different topic. The researchers al- 
so consulted with representatives of the Government Accounting Office (GAO), which was 
planning to survey secondary state directors in the first quarter of 2009 to assess field per- 
spectives on Perkins IV performance and accountability systems. Because this study was be- 
ing conducted at the request of Congress, it was not possible to delay or cancel it; instead, 
the researchers consulted with GAO staff and, where appropriate, redesigned this study’s 


sutveys to minimize overlapping questions. 


? Each state designates a state director of CTE with primary oversight of the federal grant. Typically, 
this person is located at either a state secondary or postsecondary education agency, board, or com- 
mission, though other administrative structures exist. To ensure the collection of comprehensive data, 
researchers developed and administered separate surveys to the state director and the person charged 
with CTE oversight at the counterpart secondary or postsecondary education agency. 
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Survey development began in December 2008. The researchers developed drafts of four 
survey instruments, two targeting state directors with responsibility for overseeing federal 
grant activities at the secondary and postsecondary levels, and two targeting local directors 
located in LEAs and IHEs. In April 2009, the researchers shared preliminary drafts of the 
survey instruments with Department staff. The researchers also held a listening session at the 
April 2009 National Association of State Directors of Career and Technical Education 
(NASDCTEc) annual meeting to solicit feedback from state directors on key issues affecting 


Perkins IV implementation. Feedback was incorporated into the final survey instruments. 


The survey instruments were weighted toward collecting information on POS, using an orig- 
inal set of questions to capture states’ design and implementation strategies. Nearly all ques- 
tions were constructed as multiple choice, and many offered respondents the option of 
answering “Don’t Know” or “Other”, when appropriate. For many questions, respondents 
choosing the “Other” category were asked to explain their response. Where appropriate, 
state directors were asked to upload electronic copies of state guidance and policies, as well 


as to answer incomplete questions before terminating the survey. 


Four former state directors (two secondary and two postsecondary) and four acting local di- 
rectors (two LEA and two IHE) from eight states—chosen for their geographic, size, and 
financing characteristics—pilot tested the instrument. The reviewers were selected based on 
their knowledge of the field. To promote internal consistency, a secondary state director and 
LEA director in one state and a postsecondary state director and IHE director in another 
state assessed whether cross-sector conditions within a state might affect survey interpreta- 


tion. 


To guide the review of the instrument, the pilot test volunteers received a standard set of 
questions that asked about the time required to complete the surveys; their clarity of purpose 
and organization; the surveys’ relevance to the implementation of Perkins IV; perceived ac- 
cessibility of information needed to answer the questions; and any improvements needed to 
strengthen the instrument. Reviewer feedback was used in final revisions to the survey in- 
struments. Key changes included (1) incorporating web links to permit respondents to pro- 
vide references to legislative language; (2) refining wording to clarify the meaning of 
questions; and (3) simplifying reporting for questions on the proportion of students partici- 


pating in POS. 


Survey administration 
Survey administration began in late October 2009. State and local directors received three 
mailings, with the first including a cover letter, credentials for accessing the web survey (an 
ID and password), and a brochure describing the survey. This packet also included a letter 


encouraging participation signed by the executive directors of three leading CTE stakeholder 
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organizations: NASDCTEc, the National Research Center for Career and Technical Educa- 
tion (NRCCTE), and the Association for Career and Technical Education (ACTE). 


The study goal was to achieve a unit response rate of at least 85 percent for each of the four 
populations surveyed (1.e., secondary and postsecondary state directors and LEA and IHE 
local directors).? Nonrespondents received a reminder postcard 7—10 days after the initial 
mailing. The research team began follow-up efforts immediately after the first mailing, call- 
ing potential respondents at approximately 4-, 8-, and 12-day intervals. A third and final sur- 
vey mailing included a print questionnaire and a stamped self-addressed return envelope. 
Telephone follow-up continued throughout the survey period, which was extended from its 
initial closing in mid-December 2009 through March 2010. 


Response rates: State directors 
In keeping with past national assessment procedures, this study included a universe survey of 
state directors. Surveys were administered to secondary and postsecondary state directors 
with responsibility for overseeing CTE programs within states and outlying areas.* Of the 57 
sutveys distributed at the secondary education level, 51 were returned (from 50 states and 1 
outlying area). At the postsecondary level, 48 state surveys were returned; outlying areas de- 
clined to participate.> An item response rate analysis indicated that, with few exceptions, 
state director responses exceeded the 85 percent item response rate guideline established by 
NCES. Any question for which the state director response rate fell below the 85 percent re- 


sponse rate threshold is noted in the report. 


Response Rates: LEA and IHE directors 
The researchers constructed sampling frames using the Common Core of Data (CCD) file to 
identify eligible LEAs and the Integrated Postsecondary Education Data System (IPEDS) to 
identify IHEs. Local directors were recruited to participate in the survey through a letter sent 


to the organizational leader, typically a public school district superintendent or community 


3. NCES has established a unit response rate threshold of 85 percent for surveys of the type used in 

this study. Surveys that achieve this minimum level are more likely to be representative of the target 
population they are intended to assess. See http://nces.ed.gov/statprog/2002/std2_2.asp for more 
information. 

4Perkins IV defines a “state” as “each of the several States of the United States, the District of Columbia, 
the Commonwealth of Puerto Rico, and each outlying area” (Sec. 3). The term “state” as it is used in this 
report includes the 50 states of the United States, the District of Columbia, the Commonwealth of Puerto 
Rico; and the outlying areas of the United States Virgin Islands, Guam, American Samoa, the Common- 
wealth of the Northern Mariana Islands, and the Republic of Palau. 

> This unit response pattern and rate parallels those found in the 2004 NAVE, which reported that 
complete survey and fiscal data were obtained from 45 states and that the territories declined to par- 
ticipate in the study. If the analysis were based solely on the 50 states and District of Columbia, the 

2011 study achieved a unit response rate of 96 percent for the secondary state director survey and 94 
percent for the postsecondary state director survey. 
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college president, with instructions that the survey be forwarded to the individual designated 


as the on-site Perkins IV coordinator or CTE contact person. 


SAMPLING FRAME: LEA DIRECTORS 

The target population for the LEA survey consisted of those agencies that are open, affiliat- 

ed with a public school, offer 12th-grade instruction, enroll at least one 9th-, 10th-, 11th-, or 
12th-grader, and are located within the 50 states, the District of Columbia, and Puerto Rico. 

LEAs operated or supported by the Bureau of Indian Affairs (BIA) also were included in the 
target population. 


Eligible LEAs were stratified according to the same criteria used by states to allocate federal 
Perkins IV Title I funds to local providers. Specifically, Perkins IV stipulates that 30 percent 
of Title I funds be allocated based on the number of youth ages 5-17 residing within LEA 
boundaries and 70 percent be allocated based on the number of youth ages 5-17 from fami- 
lies below the poverty line, compared to the total number of youth within each category who 
reside within participating LEAs statewide. Accordingly, the sampling strata for the LEA 
population was based on the cross-classification of agency K-12 membership and the per- 


centage of K-12 membership eligible for free or reduced-priced school lunches.’ 


To ensure that the sample distribution of LEAs reflected the population distribution of 
LEAs, the sample was allocated proportionally to the sampling strata, in which the propor- 
tion was based on the population proportion (Exhibit A.2). Assuming that the population 
proportion under the null hypothesis is 0.5, a required sample size of 1,136 LEAs from a 
population of 12,518 LEAs achieved 81 percent power to detect a difference of 0.04 using a 
two-sided binomial test. The target significance level was 0.05, and the actual significance 
level achieved by this test was 0.0499. To account for anticipated nonresponse, the required 
sample size was increased. Based on the expected 85 percent response rate for LEAs, the 
nonresponse adjusted sample size was increased to 1,337 LEAs. Because not all LEAs eligi- 


ble to participate in Perkins IV actually do so, the nonresponse adjusted sample size also was 


6 The federal Perkins IV allocation also includes Title II Tech Prep resources, which are allocated by 
states to LEAs and IHEs either through a competitive process or by state formula. These funds con- 
stitute less than 10 percent of the total state Perkins IV allocation. Because state allocation criteria used 
to distribute Title II resources may differ substantially across states, the sample selection methodology 
is based on the distribution formula for Title I funds. 

7 High school students qualifying for free or reduced-price school lunches are likely undercounted, 
because older students are less likely to take advantage of this program. Therefore, LEAs made up of 
high schools alone may qualify for less Perkins IV fanding than that for which they might otherwise 
be eligible. Because this is a systematic bias (i.e., intrastate Perkins IV funds distribution also is affected 
by this condition), there should be no effect on the sampling frame. 
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increased.’ Based on an expected 67 percent eligibility rate for LEAs identified in the CCD, 
the sample size was increased to 1,993 LEAs. Adding in the LEAs in the certainty stratum, 
the total sample size was 2,041 LEAs. The sample was selected using a stratified simple ran- 


dom sample.° 


Among LEAs, there is a subset of institutions, called area CTE centers, which provide special- 
ized CTE services on a part-time basis to students receiving most or all of their academic in- 
struction at their home high school. These centers typically serve multiple high schools and 
are administered as stand-alone schools within an LEA. A handful of these centers, however, 
function as independent LEAs. The LEA sample contained a certainty stratum for the area 
CTE centers. There were 48 LEAs in this stratum.!° 


UNIT RESPONSE RATE 

The LEA survey had a 77 percent weighted unit response rate (Exhibit A.3). This rate was 
somewhat below the 85 percent unit response rate threshold used by NCES when conduct- 
ing assessments.!! Weighted unit response rates varied among sampling strata, with only two 
strata exceeding the NCES 85 percent response threshold. An item response rate analysis in- 
dicates that, with few exceptions, LEA director responses exceeded the 85 percent item re- 
sponse rate guideline established by NCES. Any questions for which the item response rate 
fell below the 85 percent threshold are noted. 


8 Although states are required to report annually on their allocation of funding among educational sec- 
tors and administrative programs, they are not required to document their number of local subgrants 
to the Department. The most current statistics on LEAs and THEs participating in Perkins come from 
the 2004 NAVE, which suggested that roughly two-thirds of eligible LEAs and four-fifths of eligible 
ITHEs were awarded Perkins funds in the 2000-01 program year. 

° While probability proportional to size sampling was considered, with the measure of size being the 
number of students at the school, it was unclear how correlated these two variables would be. There is no 
information on the frame for the number of students in CTE. This is one of the questions in the survey. 
10 In developing the area CTE center stratum, the researchers identified 48 stand-alone facilities, based 
on a review of data contained in Table 2.5 of the NCES publication Career and Technical Education in the 
United States: 1990 to 2005 (Levesque et al. 2008). [Accessed September 27, 2010, 
http://nces.ed.gov/pubs2008/2008035.pdf]. To identify stand-alone facilities, the researchers com- 
pared the names of identified centers within states to the list of centers contained in the 2001-02 
CCD. The 48 area CTE centers identified for this study were listed as independent LEAs with their 
own NCES ID number. Remaining area CTE centers were listed as schools within LEAs. These cen- 
ters were excluded from the study because information on their operations would be obtained from 
the program director of the LEAs in which they were housed, assuming that the LEAs were randomly 
selected for study participation. Subsequent to the study, researchers discovered that there were, in 
fact, 386 stand-alone area CTE centers, of which 72 were included in the sampling frame. As such, the 
actual sample included in the study covered roughly 19 percent of the actual universe. 

1! See http://nces.ed.gov/statprog/2002/std2_2.asp for NCES Statistical Standards (U.S. Department 
of Education 2002). 
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Exhibit A.2. 
Population and sample counts for LEAs, by LEA size and poverty level: 

2006-07 

LEA size 
Poverty Small Medium Large Unknown Total 
level Pop. Sample Pop. Sample Pop. Sample Pop. Sample Pop. Sample 

Total 3,261 519 5,863 934 3,220 513 174 27 12,518 1,993 

Low 546 87 1,588 253 1,057 168 0 0 3,191 508 
Medium 1,308 208 2,586 412 1,155 184 0 0 5,049 804 
High 1,200 191 1,600 255 986 157 0 0 3,786 603 
Unknown 207 33 89 14 22 4 174 27 492 78 


Exhibit reads: The stratification of LEAs based on the relative size and poverty levels indicates that of the 12,518 eligible LEAs, a 
random sample of 1,993 LEAs was identified to participate in the survey. 

NOTE: LEAs with 499 or fewer students were classified as small LEAs; those with 500 to 2,999 students were classified as medium 
LEAs; and those with 3,000 or more students were classified as large LEAs. Low-poverty LEAs are those with less than 25 percent 
of students who are eligible to receive free or reduced-price lunches; medium-poverty LEAs are those with between 25 and 50 
percent of eligible students; and high-poverty LEAs are those with more than 50 percent of eligible students. 

SOURCE: Common Core of Data (CCD), 2006-07. 


Exhibit A.3. 
Response rates for LEAs, by type of sampling unit and sampling stratum: 2006-07 


Weighted unit response rate (“/) 
Sampling unit 
Local education agency (LEA) 77 


Sampling stratum 


Area CTE center 93 
LEAs, enrollment 0-499 
FRL category 0.00—0.25 71 
FRL category 0.25—0.50 77 
FRL category 0.50 or more 72 
FRL category unknown 85 
LEAs, enrollment 500-2,999 
FRL category 0.00—0.25 80 
FRL category 0.25—0.50 75 
FRL category 0.50 or more 79 
FRL category unknown 82 
LEAs, enrollment 3,000 or more 
FRL category 0.00-0.25 76 
FRL category 0.25—0.50 80 
FRL category 0.50 or more 78 
FRL category unknown 100 
Size unknown, FRL category unknown 73 


Exhibit reads: Weighted response rates indicate that 77.1 percent of eligible LEAs would have responded to the survey. 
NOTE: “FRL” refers to students who are eligible for free or reduced-price lunches. 
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SAMPLING FRAME: IHE DIRECTORS 

The target population for the IHE survey consisted of postsecondary institutions that were 
open, eligible for Title I funding, serving undergraduate students, and one of three institu- 
tion types: public less-than-two-year postsecondary institutions, public two-year colleges, or 
tribally controlled colleges. Public four-year colleges, area or regional schools funded with 
postsecondary resources, adult schools, private nonprofit colleges, and other unidentified in- 


stitutions were excluded. !2 


Perkins IV postsecondary Title I funds are allocated based on the number of individuals at- 
tending an eligible institution who are Pell Grant recipients or recipients of BIA assistance, 
relative to the total number of such individuals enrolled within participating institutions 
throughout the state. The IHE sample was stratified based on the percentage of federal grant 
aid recipients within each [HE and by institution type.'? To ensure that the sample distribu- 
tion of IHEs resembled the population distribution of IHEs, the sample was allocated pro- 
portionally to the sampling strata, with the proportion of the sample in a sampling stratum 


based on the population proportion for the sampling stratum. 


Assuming that the population proportion under the null hypothesis is 0.5, a required sample 
size of 684 IHEs from a population of 1,423 IHEs achieves 81 percent power to detect a 
difference of 0.04 using a two-sided binomial test (Exhibit A.4). The target significance level 
is 0.05, and the actual significance level achieved by this test is 0.0438. To account for antici- 
pated nonresponse, the required sample size was increased. Based on the expected 85 per- 
cent response rate for IHEs, the nonresponse adjusted sample size was increased to 805 
IHEs. Because only about four-fifths of IHEs are actually awarded a grant, the nonresponse 


adjusted sample size also was increased to account for anticipated ineligible IHEs. Based on 


2 While Perkins IV defines eligible institutions to include public or nonprofit IHEs (as defined in Title 
IV of the Higher Education Act of 1965), LEAs and area CTE centers providing education at the post- 
secondary level, and colleges receiving BIA funding, findings from the 2004 NAVE indicate that a 
disproportionate share of federal postsecondary CTE resources flow to public less-than-two-year 
postsecondary institutions and two-year colleges (20 percent and 68 percent of all postsecondary Per- 
kins fands, respectively). Public four-year colleges and universities accounted for just 4.3 percent of 
postsecondary Perkins funds distributed in the 2000-01 program year. Although Perkins IV expands 
federal resource eligibility to include baccalaureate-granting institutions, Department staff report that 
the number of four-year colleges and universities participating in Perkins IV has remained relatively 
constant over time. Given the low likelihood that a random sample of public four-year or private 
nonprofit colleges and universities would include Perkins grantees, researchers excluded this group. 
Researchers also excluded area schools funded with postsecondary resources due to difficulties in 
identifying agencies receiving postsecondary funds. 

3 IPEDS does not distinguish Pell Grant recipients from other recipients of financial aid and is lim- 
ited to first-time, full-time freshmen. In practice, IHEs report the number of their Pell Grant and BIA 
Grant recipients to the state, which distributes resources based on submitted data. IPEDS provides 
the best standardized national data on the percentage of students within IHEs who ate eligible for 
student aid. 
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the expected 80 percent eligibility rate for IHEs, the final sample size was increased to 1,006 


THEs. The sample was selected using stratified simple random sampling. 


Exhibit A.4. 
Population and sample counts for [HEs, 
by institution type and level of financial need: 2006-07 


Institution type 


Public Public 

Level of less-than-two-yeat two-year Tribal Total 
financial need Pop. Sample Pop. Sample Pop. Sample Pop. Sample 

Total 265 187 1,126 797 32 22 1,423 1,006 
Low 28 20 189 134 3 2 220 156 
Medium 133 94 872 617 12 8 1,017 719 
High 61 43 62 44 17 12 140 99 
Unknown 43 30 3 2 0 0 46 32 


Exhibit reads: The stratification of IHEs based on institution type and the financial needs of enrolled students indicates that of the 
1,423 eligible IHEs, a random sample of 1,006 IHEs was identified to participate in the survey. 

NOTE: “Level of financial need” represents the percentage of first-time, full-time students receiving federal grant aid within an JHE, 
compared to the total population of students within the IHE. Low financial need IHEs are those with less than 25 percent of 
students receiving federal grant aid; medium financial need IHEs are those with between 25 and 75 percent of such students; and 
high financial need THEs are those with more than 75 percent of such students. 

SOURCE: Integrated Postsecondary Education Data System (IPEDS), 2006-07 and 2007-08. 


UNIT RESPONSE RATE 

The IHE survey had a 91 percent weighted unit response rate (Exhibit A.5), which exceeded 
the 85 percent response rate threshold established by NCES. Weighted unit response rates 
varied among sampling strata, with all but two strata exceeding the 85 percent response 
threshold. An item response rate analysis indicates that, with few exceptions, [HE director 
responses exceeded the 85 percent item response rate guideline. Any questions for which the 


item response rate fell below the 85 percent threshold are noted. 
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Exhibit A.5. 
Response rates for IHEs, by type of sampling unit and sampling stratum: 2006-07 


Weighted unit response rate (%) 
Sampling unit 


Institutions of higher education (HE) 91 


Sampling stratum 


Public less-than-two-year 


Financial need category 0.00-0.25 100 
Financial need category 0.25—0.75 97 
Financial need category 0.75 or more 94 
Financial need category unknown 93 


Public two-year 


Financial need category 0.00—0.25 94 

Financial need category 0.25—0.75 89 

Financial need category 0.75 or more 80 

Financial need category unknown 50 
Tribal 

Financial need category 0.25-0.75 re) 

Financial need category 0.75 or more. 100 


Exhibit reads: Weighted response rates indicate that 90.6 percent of eligible IHEs would have responded to the survey. 


Nonresponse bias analysis 
Response rates show some differential responses across the sampling strata. Because the 
sampling strata were thought to be associated with important outcome variables, differential 
response across the sampling strata can introduce bias into the estimates because of nontre- 
sponse. Consequently, the sampling stratum was one variable used in the weighted adjust- 
ment process for LEA and IHE data to minimize the potential for nonresponse bias. To 
further investigate the potential for nonresponse bias, the researchers used information gath- 


ered during the survey administration process. 


For the LEA data, the researchers used a tree-based model to investigate the level of effort 
variable, represented by the total number of comments recorded by follow-up staff, and a 
monetary reimbursement variable, based on the incentives offered to potential participants, 
which ranged from $0, $50, or $100. The total number of comments was taken from the In- 
stitutional Contacting System (ICS) and serves as a proxy for the number of telephone con- 


tacts attempted or completed for each potential respondent. 


Using the tree-based methodology, the researchers looked for the best split along the con- 


tinuum of the total number of comments or among the different incentive levels (Exhibit 
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A.6). This split would partition the data into two groups of IHEs with different response 


propensities for each group. Then within each group, the researchers used the tree-based 


methodology to look for the best split again to further differentiate the response propensities 


for each group. The tree split the LEAs based on the total number of comments at 9.5 and, 


within the highest number of comments group, by whether or not reimbursement was pro- 


vided. Therefore, the three groups of LEAs were created based on the total number of 


comments, and monetary reimbursements were as follows: a group with 9.5 total comments 


ot fewer, with a response propensity of 86.8 percent; a group with more than 9.5 total com- 


ments but no incentive, with a response propensity of 42.8 percent; and a group with more 


than 9.5 total comments and an incentive, with a response propensity of 100.0 percent. 


Exhibit A.6. 


Classification tree for LEA response propensity 


No. cases % cases 


Class 


Eligible, noninterview 322 22.6 


1,105 774 
1,427 


Interview 
Total 


Total Comments < 9.50 


Class 


Eligible, noninterview 


Class = Interview 

Class 

Eligible, noninterview 140 13.2 

Interview 919 86.8 
Total 1,059 


No. cases % cases 
Interview 


Total 


No fiscal incentive 


Class = Eligible, noninterview 


Class No. cases % cases 


Eligible, noninterview 182 rai) 
Interview 136 42.8 
Total 318 


Total Comments > 9.50 


No. cases % cases 


182 49.5 
186 50.5 
368 


Fiscal incentive provided 


Class = Interview 
Class 

Eligible, noninterview 0 0.0 
100.0 


No. cases % cases 


Interview 50 


Total 50 


Exhibit reads: Among the 1,427 eligible LEAs, 1,059 were contacted 9.5 times or fewer, with 86.8 percent responding to the survey; 
318 were contacted more than 9.5 times but not offered an incentive, with 42.8 percent responding to the survey; and 50 were con- 
tacted more than 9.5 times and offered an incentive, with 100.0 percent responding to the survey. 


APPENDIX A. METHODOLOGY 


A-13 


For the IHE data, the researchers investigated a level of effort variable—the total number of 


comments—using a tree-based model (Exhibit A.7). Using the tree-based methodology, the 


researchers looked for the best split along the continuum of the total number of comments. 


This split would partition the data into two groups of IHEs with different response propen- 


sities for each group. The tree-based methodology split the IHEs based on the total number 


of comments at 7.5 comments. Therefore, the two groups of IHEs created based on the to- 


tal number of comments were a group with less than or equal to 7.5 total comments, with a 


response propensity of 95.7 percent, and a group with greater than 7.5 total comments, with 


a response propensity of 75.0 percent. 


Class 


Eligible, noninterview 


Exhibit A.7. 
Classification tree for IHE response propensity 


Interview 
Total 


Total Comments S 7.50 


Class = Interview 
Class No. cases 


Eligible, noninterview 28 


Interview 623 


Total 651 


% cases 
43 
95.7 


No. cases % cases 


81 9.4 
782 90.6 
863 


Total Comments > 7.50 


Class = Interview 

Class No. cases % cases 

Eligible, noninterview 53 250) 

Interview 159 75.0 
Total 212 


Exhibit reads: Among the 863 eligible IHEs, 651 were contacted 7.5 times or fewer, with 95.7 percent responding to the survey, and 
212 were contacted more than 7.5 times, with 75.0 percent responding to the survey. 


Nonresponse adjustment factor 


The nonresponse adjustment was a ratio adjustment within each weighting class created by 


the cross-classification of the stratum variable and the groups identified by the nonresponse 


bias analysis (i.e., two groups for the IHE data and three groups for the LEA data). In each 


weighting class, the nonresponse adjustment factor is the sum of the design weights for the 


eligible cases in the weighting class divided by the sum of the design weights for the re- 


sponding cases in the weighting class. That is, the nonresponse adjustment factor for the 9” 


weighting class, dy, is 
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where gis the weighting class, EF, is the set of eligible cases in the g” weighting class, C; is the 


set of responding cases in the g” weighting class, and d; is the design weight for the 7 case. 


Analysis weight 
The analysis weight, or nonresponse adjusted weight, is the product of the nonresponse ad- 
justment factor and design weight for responding cases and zero for the eligible nonrespond- 


ing cases. That is, the analysis weight, », is 


oe ° Gi» responding cases 
= 


0, nonresponding cases 


where a, is the nonresponse adjustment factor for the g’ weighting class and d; is the design 


weight for the / case. 


To account for the stratified simple random sample design and differential weighting, point 
estimates and standard errors for descriptive statistics of categorical and continuous variables 
were calculated. All comparisons of the IHE and LEA survey data were tested for statistical 
significance using the Student’s éstatistic, and all differences cited are statistically significant 
at the p < .05 level. 


State Fiscal Allocation Surveys 


The researchers collected Perkins IV FY 2009 allocation data from eligible agencies to assess 
state distribution of federal resources to LEAs and IHEs and to extend trend analyses te- 
ported in the 2004 NAVE." To collect data, the research team sent secondary and postsec- 
ondary state directors a letter containing instructions for compiling allocation records and a 
requested file format for electronic submissions. States using an alternative formula to allo- 
cate funds were asked to send the allocation formula or criteria along with their state alloca- 


tions. 


State directors also were asked about their uses of funds for state administration and leader- 
ship purposes. Comparison of these data with those maintained by the Department revealed 


some differences in states’ reported uses of funds. After accounting for rounding and errors 


'4 Data for the 2008-09 program year correspond to the second year of Perkins IV. Previous NAVE 
studies reported on data for FY 1992, corresponding to the first year of Perkins II (1990 reauthoriza- 
tion), and on data for FY 2001, corresponding to the second year of Perkins III (1998 reauthorization). 
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in state submissions, the researchers identified differences ranging from a few thousand to a 
few hundred thousand dollars in roughly one-third of states. After consulting with state di- 
rectors and the Department, the research team determined that federal administrative data 
offered the most accurate depiction of the uses of funds by eligible agencies for state-level 
activities, and state-reported data was most appropriate for assessing states’ allocation of Ti- 
tle I funds to LEAs and IHEs. 


Response rates 
The study goal was to achieve a survey participation rate of 100 percent. The fiscal data re- 
quest was mailed approximately two weeks following the initiation of the online state ditec- 
tor surveys. In addition to reminder mailings and phone calls during the online study effort, 
the researchers followed up with individual state directors, where necessary, to encourage 
their participation in the fiscal analysis. File submissions were made electronically via email, 


with supporting documentation provided in electronic or hardcopy formats. 


Of the 52 surveys distributed to secondary state directors, 51 were returned with fiscal allo- 
cation records documenting Title I resource allocations. !5 Of the 52 surveys distributed to 
postsecondary state directors, 51 surveys were returned. A total of 50 fiscal allocation rec- 


ords documenting Tech Prep funding were returned. 


While the researchers successfully secured high unit response rates, not all states provided 
complete data for all fields. For example, some states were unable to provide information on 
the number of LEAs or IHEs participating in consortia. When this occurred, the researchers 
deleted these states for the given analysis, noting the deletion in the notes on the accompa- 
nying exhibit. To replicate trend analyses published in the 2004 national assessment, the re- 
searchers also deleted states for indicated exhibits that had not responded in the 1992 and 
2004 study years. 


Case Studies 


The study team conducted site visits in six states to collect quantitative and qualitative data 
on state and local experiences in developing finance, accountability, and POS systems. The 
six participating states were recommended by national experts and selected to provide diver- 
sity in size, geography, and organizational structure. States also were selected based on ex- 
perts’ perceptions of their relative success in implementing POS, with two states judged as 
“advanced,” two states as “average,” and two states as “lagging.” Finally, to assess changes 
across the Car/ D. Perkins Vocational and Applied Technology Education Act of 1998 (Perkins IID) 
and Perkins IV, the researchers selected two states that had been visited as part of the 2004 


15 Surveys were distributed to the 50 states, the District of Columbia, and the Commonwealth of 
Puerto Rico. Outlying areas were not included in this study component. 
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national assessment. Participants were assured that all information gathered would remain 
confidential, and any examples or quotations included in the final report would not be di- 


rectly attributed to any state or individual. 


Each site visit began with an interview with the state directors at the secondary and postsec- 
ondary education levels to gather their perspectives on the state’s progress in implementing 
Perkins IV and any challenges encountered in the process.!° The researchers subsequently 
met with state staff members responsible for the accountability and finance provisions of 
Perkins IV, as well those responsible for the design and implementation of POS. Researchers 
also visited three randomly selected local POS partnerships to collect data at the secondary 
and postsecondary levels. For case study purposes, a partnership was defined as a school dis- 
trict or area CTE center and its postsecondary affiliate. Information obtained at the state 
level was used to provide both context and a knowledge base for the local community site 
visits. In addition, researchers collected feedback from business, industry, and labor repre- 


sentatives at the local level. 


The research team used a standardized set of protocols, tailored to the roles and responsibili- 
ties of state and local administrators, to ensure the collection of consistent data across sites. 
(These protocols are presented at the end of this Appendix.) Before initiating site visits, the 
research team held a full-day training session for case study visit teams to review the purpos- 
es of the study, to train researchers on case study methodology and facilitation skills, and to 


role-play the use of the protocol at the state and local levels. 


Case study teams were composed of two or three site visitors, one of whom served as the state 
lead and the others as co-facilitator or note-taker. State and local visits typically were conduct- 
ed over the course of two—or, in one case, three—separate trips, and four senior staff with 
extensive case study experience were assigned to cover the six states. Each served as the team 
leader on all visits to one state and the local communities within it. Thus, team leaders were 
the “constant” on all visits to their states, working closely with state and local administrators 


to develop site visit agendas and arrange interviews with key respondents. 


Each case study visit also was digitally recorded and transcribed. A set of guidelines for note 
transcription and report writing were developed to structure and coordinate information 
processing. All case study team visitors also met via teleconference immediately following 
the first case study visit to debrief on the use of the protocol and field experiences. Subse- 
quently, team members held bi-weekly calls to review field experiences and share recom- 


mendations and lessons learned. Site visits were conducted between March and May 2010. 


'6 State-level visits typically occurred over two days, with researchers returning to the state to conduct 
a one-day visit at each local provider. 
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Expert Panel Review 


An expert panel assessed the quality of guidance and materials developed by states to assist 
local agencies in their POS design and implementation efforts. The expert panelists were se- 
lected based on their knowledge of POS development and implementation, nominations 
from Department staff and education stakeholders, and research team members’ knowledge 
of the field. During a series of teleconference calls, panelists’ shared ratings of state guidance 


and materials. For more information about the expert panel’s findings, see Appendix B. 


Expert panelists 
Stacy Edds-Ellis 
Director, Discover College 
Owensboro Community & 


Technical College, Kentucky 


Mark A. Elgart 
President/CEOAdvancED* 


Donna Elmore 


Associate Vice President 


Orangebure-Calhoun Technical 


College, South Carolina 


Kimberly Green 

Executive Director 

National Association of State 
Directors of CTE 


Ray Hasart 
Regional CTE Coordinator 
High Desert ESD, Oregon 


Expert panelists’ roles and responsibilities 


Tom Houlihan 
Consultant 
International Center for Leadership 


in Education, Inc. 


J.D. Hoye 
President 


National Academy Foundation 


Alisha Dixon Hyslop 
Assistant Director of Public Policy 
Association for Career and 


Technical Education 


Guy Jackson 
CTE Program Consultant 
Wyoming Department of Education 


Pradeep Kotamraju 
Deputy Director 
National Research Center for Career 


and Technical Education 


State materials were reviewed independently by two panelists, randomly paired for each state 


assignment, with each panelist reviewing materials from 10 states. To structure their work, 


the panelists received a rubric with criteria for evaluating state POS guidance, based on a list 


of 10 components of high-quality POS issued by the Department in January 2010 (e., the 


POS Design Framework). 


An initial webinar was held to train the panelists on the use of the scoring rubric. The panel- 


ists were instructed to (1) categorize identified documents and websites as relating to one or 
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more of the 10 components in the Department’s design framework; (2) provide a justifica- 
tion for this categorization; and (3) rate the usefulness of the document in guiding local CTE 


providers to develop and implement POS. 


The panelists were directed to avoid ranking states when using the rubric; review activities 
instead were intended to determine whether states had addressed each of the components 
and to assess the perceived quality and usefulness of the materials. Panel members also were 
asked to identify specific documents, tools, and resources developed by states that could 
setve as models for each component of the design framework. Following their review, the 
panelists submitted their scoring commentary to the researchers, who compiled ratings and 


feedback before sharing them with the group. 


During five facilitated webinars in May 2010, the panelists shared their completed rubrics 
and discussed their rationale for assigning ratings. Each webinar was audio-recorded and 
transcribed. Following the webinars, the researchers analyzed the panelists’ ratings, using the 
scoring rubrics and webinar transcripts to distill common themes and develop a set of pre- 
liminary findings. These themes and findings were presented to the expert panelists for 
feedback during two conference calls in June 2010. Draft results then were revised based on 
the panel’s feedback. A final webinar was held to review the final study report and debrief 


panel members. For detailed expert panel findings, refer to Appendix B. 


Collection of POS materials 
In March 2010, each state director received an e-mailed letter, co-signed by the study direc- 
tor and the chair of the expert panel, explaining the purpose of the expert panel review and 
requesting POS guidance materials. The letter referred to the information requested as part 
of the state director survey and offered an opportunity for states to provide any materials 
they had not included as part of their survey response. The letter also requested contact in- 
formation for the staff responsible for sending these materials. Several follow-up emails and 
phone calls were made either to the state director or the designated staff person in April 
2010 to collect missing materials. Twenty-three states responded by sending documents or 


website URLs that could be used to access the requested materials. 


In addition, the researchers reviewed websites maintained by secondary and postsecondary 
state agencies in all 50 states to identify guidance and materials posted for public use. State 
information—including documents submitted by states, website URLs, and other corre- 
spondence to the researchers—was distributed to the expert panel for review, along with the 


rubric template and instructions. 
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Extant Data Sources 


To take advantage of existing information sources, the researchers accessed Department 
administrative databases to obtain historical data on state’ use of Perkins IV resources. This 
study is the first to make use of these online databases, which did not exist when the 2004 
national assessment was conducted. The researchers also conducted a series of informal in- 
formation-gathering activities. These included conducting online searches of state and stake- 
holder websites, reviewing the existing CTE research literature, and seeking input from the 
CTE field. 


Department Administrative Databases 


Technological advances have changed how the federal government collects, administers, and 
stores state data submitted for annual Perkins-reporting purposes. Earlier national assess- 
ments of CTE relied primarily on nationwide surveys of state directors to compile fiscal data 
on states’ uses of federal funds for state administration, leadership, and other activities. 
These data are now available through a secure website.!’ Because states are not required to 
report on the allocation of Title I funds to local providers, federal databases could not be 


used to assess local allocations. 


States submit fiscal and accountability data to the Department annually, using a web-based 
submission instrument. The Department maintains the data online, in a password-protected 


website. !8 


Online Searches 


The research team members reviewed state websites and the online literature to identify 
documents and materials describing states’ design and implementation of Perkins IV finance, 
accountability, and POS systems. This task included accessing state five-year plans for Per- 
kins IV, along with state websites. The researchers also reviewed websites of professional as- 
sociations and research groups, such as the NASDCTEc, NRCCTE, and ACTE. 


Field Communications 


To assess perspectives from the field, the researchers participated in a series of listening ses- 
sions with state directors and their staff at several national meetings. This included sharing 


information on the study with state directors at the spring 2009 annual NASDCTEc confer- 


'7'The researchers accessed the Perkins Database located at www.perkinscar.com (accessed September 
27, 2010) to obtain information submitted by states as part of their annual state plan submissions and 
states’ uses of federal fiscal allotments. 

18 The researchers accessed annual finance data from the Perkins CAR database available at 
www.perkinscar.com (September 27, 2010). 
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ence and the Department’s Data Quality Meeting in December 2009. The researchers also 
presented updates on study progress and solicited input from state directors and administra- 


tors as part of Department-sponsored quarterly conference calls in May 2009 and July 2010. 
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State Director Survey—Secondary 


SU 
ew NACTE Reference copy only. Please complete 
Peerage blak boone the survey on the web. 


© Welcome/Contact_info 


Welcome to the Web-based National Assessment of Career and Technical 
Education survey (NACTE) sponsored by the U.S. Department of Education. 


Please take a moment and complete the contact information below before 
proceeding with the survey. 


First Name: | 
Last Name: l..... 
Title: .... 
Phone (XXX-XXX-XXXX):[—O—~—~—S—S—SOSCS 
Email: 
Address: CL] 


Address 2 (optional): 
City: 
State: 


Zip code: 


© Welcome/Confidentiality 


Uses of the Data: 

This study is being conducted for the U.S. Department of Education, which has contracted 
with MPR Associates, Inc., and RTI International (RTI). The purpose of this study is to collect 
information relating to the implementation of the Carl D. Perkins Career and Technical 
Education Act of 2006 during the 2008-09 academic year. Because a limited number of 
institutions have been selected, we would very much like for you to participate on behalf of 
your educational institution. 


Confidentiality 

As a matter of policy, The U.S. Department of Education is concerned with protecting the 
privacy of participants in voluntary surveys. 

We want to let you know that: 


Your responses will be merged with those of other respondents and will not be 
identified as the agency you represent, except as required by law. 

You may skip questions you do not wish to answer; however, we encourage you 
to answer as many questions as possible, because incomplete data will reduce the 
value of the information provided to Congress. 

Your answers may be used only for statistical purposes. Participation is voluntary 
and will not affect any aid or other benefits that your state agency, LEA, or 
institution may receive. 


If you have questions about the study, you may contact: 


the NACTE Help Desk at (866) 434-4718 

Steven Klein, the study's director, at (503) 963-3757 or sklein@mprinc.com, or 
Michael Fong, the U.S. Department of Education's Project Officer, at (202) 401-7462, 
Michael. Fon d.gov. 


If you have questions about your rights as a study participant, you may contact RTI's Office of Research 
Protection toll free at (866) 214-2043. 
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© Welcome/Definitions 
Definitions: 


Some questions in the survey include a hyperlink (a word or group of words in blue font that is underlined) 
that, when clicked on, brings up a definition. 


For example, 
Career and Technical Education (CTE): The term ‘career and technical education (CTE) means 
organized educational activities that— 

(A) offer a sequence of courses that— 

(i) provides individuals with coherent and rigorous content aligned with challenging academic standards 
and relevant technical knowledge and skills needed to prepare for further education and careers in current 
or emerging professions; 

(ii) provides technical skill proficiency, an industry-recognized credential, a certificate, or an associate 
degree; and 

(ili) may include prerequisite courses (other than a remedial course) that meet the requirements of this 
subparagraph; and 

(B) include competency-based applied learning that contributes to the academic knowledge, higher-order 
reasoning and problem-solving skills, work attitudes, general employability skills, technical skills, and 
occupation-specific skills, and knowledge of all aspects of an industry,including entrepreneurship, of an 
individual. 


Programs of Study (POS): Programmatic coursework that meets the following: incorporates both 
secondary and postsecondary education elements; includes coherent and rigorous content aligned with 
challenging academic standards and relevant career and technical content in a coordinated, non- 
duplicative progression of courses that align secondary education with postsecondary eduction to 
adequately prepare students to succeed in postsecondary education; may include the opportunity for 
secondary education students to participate in dual or concurrent enrollment programs or other ways to 
acquire postsecondary education credits; and leads to an industry-recognized credential or certificate at 
the postsecondary level, or an associate or baccalaureate degree. 


© Welcome/Navigation_Instructions 
Instructions and Navigation: 
NACTE Help Desk: 866-434-4718 


To navigate through the survey you will notice two buttons in the lower left hand corner of each screen titled 
“Previous” and “Next”. You can select either button at anytime to move through the survey. 


Alternately, you can also use the navigation panel located on the left-hand side of the screen. You may jump 
to any question in navigation panel simply by clicking on the question number. 


The questions in the navigation panel will turn green once they have been viewed/answered and 
have passed preliminary validation. 

If a question does not pass preliminary validation either because of incomplete information or 
errors with the response, the question in the navigation panel will turn yellow. 

Questions in the navigation menu that are grey, and have a strike through are not applicable to 
you (e.g., Q62). 


The “Save & Exit” button located in the lower right-hand corner of the screen allows you to exit the survey at 
any time. The next time you login to the survey you will return to the same question. 


Also, located in the lower right-hand corner of the screen is the “Help” button. It contains information 
regarding recommended browsers, and contains definitions for the survey response items (i.e., drop-down 
boxes, checkboxes, and hyperlinks). 


At the end of the survey, please select the “validate” button located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and that there are no errors 
present in your responses. 
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© Programs of Study/Intro 


© Programs of Study/Q1 


Has your state passed legislation supporting the development and/or 
implementation of Programs of Study (POS) since the reauthorization of the Carl D. 
Perkins Career and Technical Education Improvement Act of 2006 (Perkins)? 


© Yes, in response to the 2006 Perkins reauthorization 

Cc Yes, for reasons other than the 2006 Perkins reauthorization 
© No, already have supporting legislation in place 

© No 

© Don’t know 


© Programs of Study/Q1_upload 


Please attach a copy of the legislation supporting the development and/or 
implementation of POS or provide web link. 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, .pdf. If you have some other file 
type, please call our help desk at 866-434-4718. 


Please attach a copy of the legislation or provide a web link: I 


LE Do not have electronic copy of legislation 


© Programs of Study/Q2 

Has your state modified or expanded its administrative policies supporting POS 
development and/or implementation since the reauthorization of the 2006 Perkins 
Act? 


© Yes, in response to the 2006 Perkins reauthorization 

© Yes, for reasons other than the 2006 Perkins reauthorization 
© No, already have supporting legislation in place 

© No 

© Don’t know 
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© Programs of Study/Q2_upload 
Please attach a copy of the policies supporting POS development and/or 
implementation or provide web link 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, .pdf. If you have some other file 
type, please call our help desk at 866-434-4718. 


Please attach a copy of the policies or provide a web link: | 


TC Do not have electronic copy of legislation 


© Programs of Study/Q3 


Is state approval required for all POS offered within Local Educational Agencies 
(LEAs)? 


© Yes, state approval is required for all POS 

© No, state approval is only required for a subset of POS 

© No, state approval is required only for the one (1) POS required of each local agency 
© No, state approval is not required 

© Don’t know 


© Programs of Study/Q4 
Does state administrative policy require that all career and technical education 
(CTE) programs in your state eventually become POS? 


© Yes, during the lifetime of the Perkins Act of 2006 
© Yes, no timeline specified 

© No 

© Don't know 


NACTE State Director Survey- Secondary Page 4 of 46 


= Reference copy 
NACTE Please complete 
= ee ee aan Sewanee 


© Programs of Study/Q5 
How many state-approved POS offered in your state in the 2008-09 program year 
were developed either at the state agency or LEA level? 


Enter number (whole numbers only, do not include commas or other non-numeric characters), mark 


ces 


"Don't Know" if you do not have this information 


State developed POS 


State-approved, locally developed POS, using state template or 
guidance 


Pataetesenteurneddeadeedacadesdu test esuanasempiuarces cdandencessce'ssueacmseanacwas divin Samsclntpacwaeweedscloaaceaad 
State-approved, locally developed POS (i.e., no state guidance ‘ ‘ 
provided to LEA) 


© Programs of Study/Q6 


What percentage of secondary students participating in CTE in your state were 
enrolled in coursework that was part of a state-approved POS in the 2008-09 
program year? 


© None 

© Less than 25% 

© Between 26 to 50% 
© Between 51 to 75% 
© Between 76 to 99% 
© all 

© Don't know 
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© Programs of Study/Q7 


In the following table, please list the five (5) POS with the highest enrollments 
in your state in the 2008-09 program year and the number of students enrolling in 
each. If you have less than five POS then please list them all. Please do not 
include any POS that did not receive state approval. 


Type in the name of the POS (or name that best describes the POS offerings at the local level) in the 
first box, code it into the corresponding Career Cluster area (or closest corresponding area) by clicking 
on the second box, and enter the number of students enrolling in the POS in the third box. Mark ‘Don’t 
Know’ if you do not have access to this information. 


Number of students 
Name of state-approved POS enrolled in coursework 
NOTE: Please crosswalk your POS to the appropriate cluster using the in the POS in the 
pull-down menu provided 2008-2009 program 
year 
| T Don't 
1: ' | ' 
[-select one-- | ' Know 
| T Don't 
2: i | : 
[-select one-- x] Know 
‘| ‘ [Don't 
ae ‘| 
[--select one-- x] ; : Know 
‘| ‘ [Don't 
met : | : 
[--select one-- x] : Know 
‘| ' ODon't 
ae) ' | ; 
[-select one-- 7] Know 
--select one-- NZ 


--select one— 

Agricultural, Food, and Natural Resources 
Architecture and Construction 

Arts, Audio/Video Technology and Communications 
Business Management and Administration 
Education and Training 

Finance 

Government and Public Administration 

Health Science 


Hospitality and Tourism 
Human Services 


Science, Technology, Engineering and Mathematics 
Transportation, Distribution and Logistics 
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© Programs of Study/Q8A 
Which of the following elements of a POS does your state require LEAs to adopt as 
part of a state-approved POS? (Note: These elements are not specifically defined 
within the Perkins legislation, but may be required by states as part of a locally 


developed POS.) 


Standards 


Align with state postsecondary standards or 


requirements for program completion. : 


Align with technical standards that were... ' c ' c ' c 
a. state-developed ' 


b. industry-developed 


Align with state work-ready standard 
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© Programs of Study/Q8B 


Which of the following elements of a POS does your state require LEAs to adopt as 
part of a state-approved POS? (Note: These elements are not specifically defined 
within the Perkins legislation, but may be required by states as part of a locally 


Uses a curriculum that spans the secondary and 
postsecondary levels 


' 1 ' 
anbnenneneenn eee knee eee eee eee eee eee eee eee eee eee Janene eeeenesnetieeeeeee eee eeninebeene ene nee, 
1 1 


Uses a curriculum that is non-duplicative across : c 
secondary and postsecondary levels ‘ 


c 


‘s 


1 . 1 
ne a ee Sonne aS eae nS a ye Sn ne yn ee 
' ' 


Is part of an articulation agreement with a c 
postsecondary institution 


c 


c 


' ' 
mm SS ooo 
' 


Is covered by a state-wide articulation agreement 


Offers secondary CTE courses that earn academic credit : 
toward HS diploma ' 


' 
a gp ef ee ee Ee a ae a ee 
1 


Offers secondary CTE courses that provide : 
postsecondary credit through dual or concurrent ‘ c 
enrollment : 


2 


c 
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© Programs of Study/Q8C 

Which of the following elements of a POS does your state require LEAs to adopt as 
part of a state-approved POS? (Note: These elements are not specifically defined 
within the Perkins legislation, but may be required by states as part of a locally 


developed POS.) 
Assessments 


Assesses technical skill attainment through... 
a. state-developed exams that are... 
(i) aligned with state technical standards 


(iv) other 


(specify) | 


e. locally developed exams that are... 
(i) aligned with state standards 
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© Programs of Study/Q8D_sec 

Which of the following elements of a POS does your state require LEAs to adopt as 
part of a state-approved POS? (Note: These elements are not specifically defined 
within the Perkins legislation, but may be required by states as part of a locally 
developed POS.) 


Credentials 


Leads to a credential with special recognition on high 
school diploma or transcript 


Other features 


Career guidance must be available 


© Programs of Study/Q9 
What action has your state taken to ensure that POS development by LEAs aligns 
with the core elements identified in the Perkins legislation? 


Mark all that apply 
I~ Creating POS models or templates 


I~ Developing curriculum guides and other materials 

I~ Designing guides for aligning CTE content with academic standards or learning outcomes 
I~ Designing guides for aligning CTE content with technical standards 

IT Requiring LEAs to provide evidence of alignment in their local plans 

I Securing secondary/postsecondary agreement on specific POS elements 

I~ Consulting the career clusters developed by the States’ Career Clusters Initiative 

I Providing training and professional development to LEA staff 

I~ Creating/created state “college ready” or “work ready” standards for graduation 


[— Other 
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© Programs of Study/Q10A_sec 


To what extent did representatives from the following stakeholder groups 
participate in state agency efforts to develop state-approved POS? 


Local chamber of commerce 


© Programs of Study/Q11 


On which of the following topics did your state introduce new technical assistance 
activities to help LEAs develop state-approved POS during the 2007-08 and/or 
2008-09 program years? 


Mark all that apply 
T POS templates or guidelines 


Tl Career clusters 

T CTE content standards 

lM CTE curriculum development guidelines 

TM Academic and CTE curriculum integration 

T Secondary and postsecondary curriculum alignment 
Tl Technical skill assessments 

Tl Career guidance and counseling 

T Data systems for monitoring student progress 

Tl Aligning standards and assessments 


TM Other 
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© Programs of Study/Q12_sec 


What types of professional development did your state offer to the following groups 
involved in the development and implementation of state-approved POS during the 
2007-08 and /or 2008-09 program years? 


Mark ot that apply for each group. If your state offered other types of professional development, specify 
the f professional development in textbox(es) in the "Other" column below. 


Other (specify) vow 
‘To: 


Secondary 
administrators : 


Postsecondary | 
administrators : 


ep eigulaaal d 
faculty 


NACTE State Director Survey- Secondary Page 12 of 46 


SU 
ey NACTE Reference copy only. Please complete 
Prenat sar bladt one the survey on the web. 


© Programs of Study/Q13 
Indicate whether you agree or disagree with the following statements that describe 
your state agency’s efforts to develop state-approved POS. 


ee eek 


State staff have 
technical expertise : 
in designing POS 


There is adequate | 
federal guidance: 
to support POS 
development 


Lack of statewide : 
technical content: 
standards has 
hindered POS 
development 


Lack of statewide : 
curricular materials : 


Lack of technical 
skillassessments : 
has hindered POS : 
development 


Continued on next page... 
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© Programs of Study/Q13 


Indicate whether you agree or disagree with the following statements that describe 
your state agency’s efforts to develop state-approved POS. 


Continued... 


Strongly | Does not 
Agree Apply 


Strongly aa 
Disagree No Opinion 
Secondary CTE 

teachers have a 

good ' 
understanding of : 


understanding of 
POS 


‘ 1 1 1 1 1 
Seer e ete eee reer esses RR Re a a es 
' ‘ ' ' ' 


Secondary CTE 
teachers and 
postsecondary 

CTE instructors 
lack the necessary | 
skills or time to 
integrate academic ' 
and technical 
standards to 
design curricula 


Secondary 
teachers and 
postsecondary 
instructors are 
cooperating to 
create POS 


1 
+ 
1 

1 
1 


administrators are : 


Secondary and 
postsecondary 
state staff are 

cooperating to 
implement POS 
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© Programs of Study/Q14 
How are CTE content standards developed for the state-approved POS offered in 
your state? 


Mark all that apply 
[7 State develops secondary CTE standards 


[7 State adopts industry-based standards 

[7 State adopts 16 career clusters standards 

I Local instructors develop content standards based on state criteria 

I Local instructors develop their own standards 

[~ Local instructors adopt standards created by industry groups 

I Local instructors consult with business or advisory council to develop or select standards 
TP No standards exist 


I~ Don't know 


© Programs of Study/Q15 
What proportion of the state-approved POS offered in your state used each of the 
following sources for CTE content standards in the 2008-09 program year? 


Locally developed CTE 
standards 


Industry-developed standards 


National standards for the 16 
career clusters 


No CTE standards are required 
or exist 
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© Programs of Study/Q16 
What share of POS approved by your state incorporated each of the following levels 


of CTE content standards in the 2008-09 program year? 


Foundation-level skills (e.g., 


1 1 1 
ietesteietiedielieiistietiateeitietietieiistatatedaieiiatisiatalateieiaiaiaiahehahaiaiaiaiaied 5 i lclectecltiatiatiatictectictatiatiath lelictieliattteteiielatetateh telieieiaiahahalieiaieieicheiah teleiaiahehehaisiieteieiaheh wieieiaieiaiahehaheieieiahed 
' ‘ ' ' 


Cluster level skills (i.e., 16 ' c ' C ' o 6 ns 
national career clusters) ' ' ' 


Pathway level skills (i.e., 79 
) 


Occupational level skills (e.g., 
job-specific) 


© Programs of Study/Q17 
How many of the state-approved POS offered by LEAs in your state used the 
following tools to assess students’ technical skill attainment in the 2008-09 


program year? 


National licensing or 
credentialing exams 


Course or program completion 
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© Programs of Study/Q18_sec 
For what proportion of state-approved POS are the following available at the 
secondary level? 


Academic credit for CTE 
coursework that may be applied 
toward graduation 


A secondary CTE certificate or ' 
credential that is noted on the 

high school transcript or 

diploma 


Postsecondary academic course ! 
credit ' 


© Programs of Study/Q19A 


How are LEAs monitored by the state to ensure they are offering state-approved 


Pos? 


Mark all that apply 
I State agency staff visit individual sites 


T LEA administrators visit individual sites 

I~ Through state reporting requirements for POS 
I~ Other approach 

I Not applicable/Not done 


© Programs of Study/Q19B 


How are POS offered within LEAs evaluated to assess whether these programs are 
accomplishing the goals for which they are intended? 


Mark all that apply 
I~ State agency staff visit individual sites 


I LEA administrators visit individual sites 
I~ Through state reporting requirements for POS 
I Other approach 


I Not applicable/Not done 
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© Programs of Study/Q20A4 
Who is responsible for monitoring the implementation of state-approved POS for 
Perkins compliance? 


Mark all that apply 
I~ State agency staff 


TM LEA administrators of CTE 

I Secondary teachers or administrators 
I Postsecondary faculty or administrators 
I Advisory committees 

I~ Other agency or individual 

I Not applicable/Not done 


© Programs of Study/Q20B 
What changes or additions to the Perkins legislation regarding POS would you 
recommend that Congress consider for the next reauthorization? 
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© Accountability/AccountabilityIntro 

The questions in the remainder of this survey address ALL of the CTE 
programs offered in your state irrespective of whether or not they qualify as 
a POS. 


© Accountability/Q21 
Overall, how important were the following issues in negotiating your state 
performance benchmarks and targets with the U.S. Department of Education? 


Somewhat 
janeons t 


State program performance 
goals for Perkins measures 


Past state performance on 
Perkins measures 


Anticipated changes in state 
conditions, excluding funding : 


Anticipated changes in CTE 
program offerings i 
Anticipated changes in student: 
populations participating in 


Anticipated change in state 
and/or local funding 


© Accountability/Q22 


How difficult was it to reach agreement when negotiating with the U.S. Department 
of Education to establish annual performance benchmarks and targets for your 


Annual FAUPL 
performance 
targets 
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© Accountability/Q23 
When negotiating with the U.S. Department of Education to establish annual 
performance benchmarks and targets for your state what were the reasons for 


encountering difficulty? 


Mark all that apply 
I Insufficient baseline data upon which to negotiate targets 


[~ Lack of clear guidance from the Department on what constitutes continuous improvement 
Il Anticipated difficulty in gathering and submitting required data 
I” Anticipated difficulty in negotiating similar performance targets with local grantees 


I~ Concern over the repercussions of the state failing to meet performance targets 


[~ Other 


© Accountability/Q24 
What percentage of LEAs in your state have negotiated performance improvement 
targets differing from your state-established targets on one or more performance 


measures in 2008-09? 


Enter percentage (whole numbers only, do not include % or other non-numeric characters) or mark 
“Don’t Know” if you do not have this information 


Enter Mark If 
eames) Estimate 


Performance levels and improvement targets are individually 
negotiated for what percentage of LEA? 
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© Accountability/Q25 
How important were the following factors when negotiating performance targets 


Not sarbhir Very 
Important pli Important 


Past performance on Perkins 
measures 


Anticipated changes in CTE 
program offerings 

Anticipated changes in crass 
populations participating in 


Anticipated changes in state 
and/or local funding 
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© Accountability/Q26_sec 
OVAE issued non-regulatory guidance to assist states in developing their 
accountability systems for the 2006 Perkins Act. How did your state make use of 


the OVAE guidance in crafting each of the following? 


Identifying Populations 


Did Not 
ee 


CTE participants 


Placement into employment, advanced 
education, and the military (5S1) 


Nontraditional completion (6S2) 
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© Accountability/Q27 


Which of the following types of technical assistance did your state provide to assist 
LEAs in implementing accountability measures and reporting during the 2008-2009 
program year? 


Mark all that apply 
l Issued guidance on data collection 


I~ Statewide training on data collection and reporting 
P Local site visits 

I Help line (telephone or email) 

D Provided individualized technical assistance 

[7 Other 


I Did not provide any technical assistance 


© Accountability/Q28 
Does the database you use for Perkins reporting purposes enable you to: 


Monitor individual students through grade 12 


Link high school records to administrative records for 
students enrolling in in-state community colleges 


Link high school records to administrative records for 
students enrolling in in-state four-year colleges or 
universities 


Link high schools records to administrative records for : 
students enrolling in out-of-state community colleges, : 
four-year colleges, or universities 


Permit you to report disaggregated performance ' 
outcomes for populations referenced in the Elementary : 
and Secondary Education Act (ESEA), also known as 

the No Child Left Behind Act 
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© Accountability/Q29 
What types of quality control measures has your state adopted to ensure the 
validity and reliability of locally reported data? 


Mark all that apply 
I~ Conduct desk audit of LEA data to identify inaccurate information 


I Use electronic error checking strategies to identify inaccurate data 


I Provides technical assistance and/or guidelines to LEAs on data collection and editing 
procedures 
[ State compares totals to prior year reports 


[~ Other 
T No quality control measures have been adopted 


© Accountability/Q30_sec 

How confident are you that the data you are collecting from LEAs for each of the 
following Perkins core indicators of performance accurately reflect local 
performance? 


Core Indicator 1 


Core Indicator 2 


Not Somewhat Very Not 
Confident Confident Confident Applicable 
rr ns rs a 


Technical 
skill ‘ 
attainment: 


Continued on next page... 
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© Accountability/Q30_sec 


How confident are you that the data you are collecting from LEAs for each of the 
following Perkins core indicators of performance accurately reflect local 
performance? 


Core Indicator 3 


Not Somewhat Very 
Confident Confident Confident 


Proficiency |! 
credential, : 
certificate, : 
or degree : 


Core Indicator 4 


Somewhat Very Don’t Not 
Confident Confident Know Applicable 


‘i : s 


‘s : c ‘ = : cS 


Placement in 
the military 


Not 
Confident 


traditional 
completion 
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© Accountability/Q31 


How confident are you that LEAs can report -- complete and accurate data for the 
following special populations, as defined by ESEA? 


Not Somewhat Very Don't Not 
Confident Confident Confident Know | Applicable 
es 5 @ |: a er rer 


with 


Individuals 
from 
economically : 
disadvantaged : 
families, ' 
including 
foster children : 


‘ ' ‘ ‘ ' ' 
sini in ce li a, a ee a a a ae ee 
1 1 1 1 ' ' 
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© Accountability/Q32_sec 

What proportion of LEAs in your state were unable to achieve their negotiated 
performance target for each of the following Perkins accountability measures during 
the 2008-09 program year? 


Academic 
attainment — : 
ile ata 
arts (1S1) 


attainment — 


Mathematics (182): 


Secondary school 
completion (381) | 


Placement in 
employment, 
higher education, : 
or the military 


Non-traditional 
completion (6S2) 


© Accountability/Q34 


How many LEAs in your state are required to develop a Perkins program 
improvement plan for 2009-10 on the basis of performance data that were generated 
in the 2008-09 program year? 


Enter number (whole numbers only, do not include commas or other non-numeric characters), mark 
"Don't Know" if you do not have this information 


Enter Number 


LEAs in need of improvement 
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© Accountability/Q35A 
In what ways and to what extent does your state currently use Perkins secondary 


rials @ | Used quite Used Used not | Used not 
Pee a bit somewhat | very much at all 
* > @ a 


To reward LEAs that are : : 


core indicator data? 


To provide targeted technical 
assistance 


To identify unusually effective | 
programs 


© Accountability/Q35B 

On the previous question you indicated your district did not use its prior year 
Perkins accountability data in 2008-09 in the following ways. Are there plans to 
use the data for these purposes in the next two years? 


To identify unusually effective programs 


AO RR ae RE RA REE RR EERE TTT SR Rae eR aE HR 
To identify special populations not being adequately —: c ; c c 
se 


1 
eee eee ee eee esses ecb er ewes SSS e bese eee ees enero res eeereeeesesssecee (eee e seer eres pees eee see eset ESesrerereress 
' ' 


Other (if specified) Cc Cc Cc 
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© Accountability/Q36 
What types of consequences do existing state policies impose on LEAs that do not 
meet one or more of their Perkins IV accountability performance targets? 


Mark all that apply 
I~ Consequences exist but are unlikely to be imposed 


I~ Develop a local program improvement plan 

I Additional accountability reporting requirements 

IT Loss of, or decrease in, state or Perkins funding 

I Restrictions or loss of flexibility in the use of Perkins funds 
Other 

TM No consequences exist 


[ Don't know 


© Accountability/Q37 
How far along is your state in the development of technical skills assessments? 


Mark all that apply 
I~ Created state-developed technical skills assessment Covering all CTE programs 


[ Created state-developed technical skills assessment Covering a subset of CTE programs 


TM Adopted commercial technical skills assessment Covering all CTE programs 


[ Adopted commercial technical skills assessment Covering a subset of CTE programs 
IT Approved list of technical skills assessments for local use 

Il In process of designing or transitioning to CTE technical skills assessment system 

I Considering development options 

Other 


IT No plans to develop an assessment system at this time 
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© Accountability/Q38 
How many of the LEAs in your state used the following tools to assess students’ 
technical skill attainment in the 2008-09 program year? 


Locally developed CTE skill 
exam(s) 


National licensing or 
credentialing exam(s) 


4 


State licensing or credentialing : 


Commercially developed exam | 
(s) (e.g., NOCTI) 


CTE course or program 
completion 


© Accountability/Q39 
What percentage of CTE concentrators reported in your state’s technical skill 
assessment measure were assessed using a state or industry developed technical 
skill assessment, national or state licensing or credentialing, or commercially 
developed exam in the 2008-09 program year? 


Less than| Between 


Industry 
developed 
technical skill : 
assessment : 


licensing or : 
credentialing : 
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© Accountability/Q40 

For each of the following data sources, indicate whether your state uses this source 
to collect placement data and, if not in use, whether state legal restrictions prevent 
your state in using this approach? 


National Student Clearinghouse 


State Unemployment Insurance wage 
records 


Federal Employment Data Exchange 
System (FEDES) 


State higher education student 
database 
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© Accountability/Q41 


How does each of the following impact your state’s capacity to report placement 
data? 


up studies of 
students who 
have left high 


adequate 
response rate 


Collecting data on : 
those employed 
out-of-state 


Collecting data on 
those attending: 
an in-state 4-year : 
public : 
postsecondary 
institution 


Collecting data on ' 
those attending: 
an out-of-state 
public or private 
postsecondary 
institution 
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© Accountability/Q41B 

Does your state agency provide LEAs with feedback on their performance on the 
Perkins accountability measures to permit them to assess their performance with 
others in the state, and if so, what type of information is shared? 


Mark all that apply 
I~ Statewide performance outcomes averaged across all LEA are shared 


[ Performance outcomes are shared, controlling for LEA characteristics similar to one another 
[7 Performance outcomes are shared for all individual LEA 
T Performance outcomes that are shared are posted on the internet for viewing 


T~ No data on LEA performance are shared 


© Accountability/Q41C 

What changes or additions to the Perkins legislation regarding state or local 
accountability would you recommend that Congress consider for the next 
reauthorization? 
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© Finance/FinanceIntro 


© Finance/Q42 

How much of your Perkins funds were allocated to the secondary and postsecondary 
sectors in your state in each of the following fiscal years? If Tech Prep is merged with 
the Basic Grant, report all funds in Title I Basic Grant. 


Enter dollar amount (whole numbers only, do not include $, commas, or 
other non-numeric characters) 


© Finance/Q43 

If your state changed the percentage split in its allocation of Title I funds to 
secondary and postsecondary education since the adoption of Perkins IV, what 
factors brought about this action? 


Mark all that apply 
[7 Change in state legislative policies or priorities 


I~ Adoption of state economic development initiative 

Il Increased focus on academics over CTE instruction at secondary level 

Il Increased focus in providing CTE instruction in one sector over the other 
I~ Change in state financing of CTE using non-federal funds 

[7 Other 

T Not applicable, state has not changed its funding split 


I~ Don't Know 
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© Finance/Q44 
Is your state planning to make any changes to the amounts of funding allocated to 
secondary and postsecondary sectors for the remainder of Perkins IV? 


c Yes, increase secondary sector share of funding 

© Yes, increase postsecondary sector share of funding 
© No 

© Don't Know 


© Finance/Q45 
How many local applications for 2008-09 program year Perkins IV, Title I funding did 
your state agency receive, and how many of these were approved or disapproved? 


Enter number (whole numbers only, do not include commas or other non-numeric characters), mark 
"Don't Know" if you do not have this information 


Total number received 


Number disapproved 


© Finance/Q46 
What were the principal reasons for disapproval? 


Mark all that apply 
I No description or inadequate description of local plans for allocating funds 


IT No description or inadequate description of local plans for collecting accountability data 
I Failed to provide accountability data in the past 


TM Poor past performance 


Il Failed to agree on negotiated performance target in the past 
I~ Program determined to lack sufficient quality 


[ Other 
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© Finance/Q47_sec 

What amount of your state’s 2008-2009 Perkins IV, Title I secondary funds went 
to each of the following types of programs? [Note: if your state consolidated its 
Title II Tech Prep funds into its Title I Basic Grant, report all funds in the Title I 


category] 


Enter dollar amount (whole numbers only, do not include $, commas, or 
other non-numeric characters), mark "Don't Know" if you do not have this 


information 


Grades 9-12 in comprehensive high schools 
Grades 9-12 in specialized facilities (such as area career 
centers, ROP, or BOCES) 


Secondary CTE in Bureau of Indian Affairs (BIA) funded 
secondary schools 


© Finance/Q48_sec 

In addition to services offered in comprehensive high schools, CTE may be offered in 
dedicated area centers or technical high schools serving one or more LEAs. Please 
indicate the number of schools or agencies in your state that offered secondary CTE 
instruction in a setting other than the comprehensive high school in the 2008-09 


academic year. 


Enter number (whole numbers only, do not include commas or other non-numeric characters), mark 
"Don't Know" if you do not have this information 
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© Finance/Q49 
For the 2008-09 program year, did your state waive the minimum allocation rule 


($15,000 per substate grantee) for any grantees? (Reference: Section 132(a) (4) and 132(c) 


(1)] 


© Yes 
© No 


© Finance/Q50_sec 
How many waivers were granted for each of the following reasons? 


Enter number (whole numbers only, do not include commas or other non-numeric characters), mark 
"Don't Know" if you do not have this information 


LEA is in a rural, sparsely populated area 


© Finance/Q52 
For the 2008-09 program year, did your state use the option to create a reserve 
fund to allocate resources to LEAs through a means other than the statutory 


formula? 


© Yes 
© No 
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© Finance/Q53 
How much money was placed into the state reserve fund for distribution to 


secondary LEAs for the 2008-09 program year? 


Enter dollar amount (whole numbers only, do not include $, commas, or other non-numeric 
characters), mark "Don't Know" if you do not have this information 


Reserve fund amount 


© Finance/Q54 
What amount of your reserve funds were awarded to LEAs located in the following 


areas? 
Enter dollar amount (whole numbers only, do not include $, commas, or other non-numeric characters), 
mark "Don't Know" if you do not have this information 


Other (specify) | 


© Finance/Q55 
How many tribally-controlled or Bureau of Indian Affairs (BIA)-funded LEAs received 


Perkins grants in 2008-09 as noted? 


Enter number (whole numbers only, do not include commas or other non-numeric characters), mark 
"Don't Know" if you do not have this information 


As an individual agency 


As a member of a consortia 
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© Finance/Q56 

Of the funds used at the secondary level, on average, what percentage of your 
state’s 2008-09 State Leadership Funds were allocated for the following required 
uses referenced in Section 124 of the Perkins Act? 

[NOTE: Percentages need not total to 100%] 


Enter percentage (whole numbers only, do not include % or other non-numeric characters) or mark 


"Don’t Know” if you do not have this information 


Assessing CTE programs 


Supporting partnerships among secondary, postsecondary, 

adult education, and other local institutions 

Serving individuals in state institutions, such as state 

correctional and institutions serving disabled students ‘ : 


Supporting programs that prepare special education students: I 
for entry into high skill, high wage, high demand occupations: 


Technical assistance for eligible recipients 


© Finance/Q57 

How much of the 10 percent State Leadership funding was made available in 2008- 
09 for secondary services that prepare individuals for nontraditional fields (i.e., of 
the not less than $60,000 and not more than $150,000 earmarked for the purpose)? 


Enter dollar amount (whole numbers only, do not include $, commas, or other non-numeric 


do not have this information 


characters), mark "Don't Know" if yo 
Enter Amount 


Nontraditional funding 
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© Finance/Q58 
What percentage of your State Leadership funds for secondary education in 2008-09 


was used to support POS development or implementation? 


Enter percentage (whole numbers only, do not include % or other non-numeric characters) or mark 
“Don’t Know” if you do not have this information 


POS implementation 


© Finance/Q59A 
Indicate whether your state has a gender equity coordinator at the secondary level 


and the job status of their position. 
© None 


© Full-time 
© part-time 


© Finance/Q59B 


Indicate whether your state has a special populations coordinator at the secondary 
level and the job status of their position. 


© None 
© Full-time 
© part-time 


© Finance/Q60 
From what funding source(s) were the gender equity and special populations 


coordinator positions funded in 2008-09? 


Mark all that apply 
I~ Perkins State Leadership funds 


I Perkins State Administration funds 
I~ State funds 


7 Other 


© Finance/Q60B 
What changes or additions to the Perkins legislation regarding finance would you 


recommend that Congress consider for the next reauthorization? 
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© Finance/Q61_INTRO 

The following juestions pertaiaitn your stats = eaicheg: ystem for secondary 

CTE. Do not include Perkins funds or resources used for your state Perkir 
atch in this section. 


© Finance/Q61 
Did your state provide categorical funding for CTE services offered in secondary 
schools in the 2008-09 program year? 


© Yes 
© No 


© Finance/Q61_upload 
Please attach state CTE funding formula or web link 


Provide your file in one of these formats: .doc, .xis, .csv, .dat, .txt, .pdf. If you have some other file 


‘pe, please call our help desk at 866-434-4718. 


Please attach state CTE funding formula 


or provide a web link: [ 


© Finance/Q62 
What percentage of the categorical statewide support for secondary CTE offered in 
your state in 2008-09 would you estimate came from the following sources: (Must 


total 100 %) 


Enter percentage (whole numbers only, do not include % or other non-numeric characters) or mark 
“Don’t Know” if you do not have this information 


Federal sources (excluding Perkins) 
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© Finance/Q63 
Does your state use performance-based allocations (e.g., rewards and/or sanctions) 


for state secondary CTE funds? 


© Yes 
© No 


© Finance/Q63_upload 


Please attach description of state performance-based funding allocation formula or 
web link 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, .pdf. If you have some other file 
type, please call our help desk at 866-434-4718. 


Please attach description of state performance-based funding allocation formula 
or provide a web link: I 
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© Tech Prep/Q64 
Did your state choose to merge its Tech Prep funds into its Title I basic grant during 
or prior to the 2009-010 program year? 


© Yes 
© No 


© Tech Prep/Q65 
What are the reasons that your state opted to merge its Tech Prep funding with its 
basic grant funds? 


Mark all that apply 
Desire to incorporate Tech Prep into all CTE programs 


I~ Due to the similarity between Tech Prep and POS 

Il Avoid data burden associated with collecting new Tech Prep measures 
I~ Difficulty identifying Tech Prep students from other CTE students 

I~ Difficulty collecting data on Tech Prep students 


I~ Other 


© Tech Prep/Q66 
Is your state planning to consolidate its Tech Prep funds into its Title I basic grant 
prior to the end of the Perkins IV Act, and if so, when? 


© State has no plans to consolidate funding 
© 2010-11 program year 

© 2011-12 program year 

© Don’t know 
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© Tech Prep/Q67 
How did your state allocate grants to local Tech Prep Consortia in 2008-09 from 


funds not merged? 


© Through a competitive application and award process 
© By formula 

© Both competition and formula 

© Other (specify) 


© Tech Prep/Q67_upload 
Please attach the guidelines or criteria used to allocate grants to local Tech Prep 


Consortia in 2008-09 from funds not merged. 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, .pdf. If you have some other file 
type, please call our help desk at 866-434-4718. 


Please attach the guidelines or criteria used. 


© Tech Prep/Q68 
How many local applications for 2008-09 program year Perkins IV, Title II Tech Prep 
funding did your state agency receive, and how many of these were approved or 


disapproved? 


Enter number (whole numbers only, do not include commas or other non-numeric characters), mark 


"Don't Know" if you do not have this information 


Total number received 


Number disapproved 
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© Tech Prep/Q69 
What were the principal reasons for disapproval? 


Mark all that apply 
I No plan submitted or inadequate plan for local allocation of funds 


T No plan submitted or inadequate plan for collection of accountability data 


I Poor past performance 
I~ Program determined to lack sufficient quality 


I~ Other 


© Tech Prep/Q70 


Which, if any, of the following obstacles does your state face in tracking Tech Prep 
students as they transition between the secondary and postsecondary education 


sectors? 


Mark all that apply 
I~ Secondary and postsecondary sectors do not share a common student identifier 


I~ Secondary and postsecondary sectors do not have a means of sharing data 
[— Secondary Tech Prep students do not self-identify (or know they are a Tech Prep student) 


when matriculating at a postsecondary institution 
Il Postsecondary institutions do not attempt to identify incoming secondary Tech Prep students 


I~ Other 
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© Tech Prep/Q71 


The 2006 Perkins introduced a new set of accountability measures for Tech Prep 
students. Indicate your level of agreement or disagreement with each of the 
following statements: 


ee No Strongly 
eel Agree 


New measures impose a 
substantial data burden on 


The new measures will support: 
program improvement efforts | 
undertaken at the state level : 


‘The new measures will support: ee a ne: a. oes oy aie 
program improvement efforts © © : © : @ + Gos GF 4 
undertaken at the /ocal level : ‘ : ‘ ‘ : 


The existence of the new ' ' ' ' ' 
measures will likely cause my 

state to merge its Title I and 

Title II funds 


My state is able to collect : 
accurate data on all measures : 


My state would like OVAE to 
issue nonregulatory guidance : 

to support my state in ' 

developing Tech Prep 

measures 


located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and 
that there are no errors present in your responses. If there are errors present, 
simply select the question from the validation summary list to take you directly to 
the question to correct your data. Once you have corrected the data, go to the 
navigation panel on the left side of the screen and select the form titled END. 


You have completed the Survey of State Directors of Secondary Career-Technical 


Thank you very much for taking the time to participate in this important study. 


If you have any questions or comments about this study, please contact Steven 
Klein (MPR) at 503-963-3757. 
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© Welcome/Contact_info 


Welcome to the Web-based National Assessment of Career and Technical 
Education survey (NACTE) sponsored by the U.S. Department of Education. 


Please take a moment and complete the contact information below before 
proceeding with the survey. 


First Name: | 
Last Name: l..... 
Title: .... 
Phone (XXX-XXX-XXXX):[—O—~—~—S—S—SOSCS 
Email: 
Address: CL] 


Address 2 (optional): 
City: 
State: 


Zip code: 


© Welcome/Confidentiality 


Uses of the Data: 

This study is being conducted for the U.S. Department of Education, which has contracted 
with MPR Associates, Inc., and RTI International (RTI). The purpose of this study is to collect 
information relating to the implementation of the Carl D. Perkins Career and Technical 
Education Act of 2006 during the 2008-09 academic year. Because a limited number of 
institutions have been selected, we would very much like for you to participate on behalf of 
your educational institution. 


Confidentiality 

As a matter of policy, The U.S. Department of Education is concerned with protecting the 
privacy of participants in voluntary surveys. 

We want to let you know that: 


Your responses will be merged with those of other respondents and will not be 
identified as the agency you represent, except as required by law. 

You may skip questions you do not wish to answer; however, we encourage you 
to answer as many questions as possible, because incomplete data will reduce the 
value of the information provided to Congress. 

Your answers may be used only for statistical purposes. Participation is voluntary 
and will not affect any aid or other benefits that your state agency, LEA, or 
institution may receive. 


If you have questions about the study, you may contact: 


the NACTE Help Desk at (866) 434-4718 

Steven Klein, the study's director, at (503) 963-3757 or sklein@mprinc.com, or 
Michael Fong, the U.S. Department of Education's Project Officer, at (202) 401-7462, 
Michael. Fon d.gov. 


If you have questions about your rights as a study participant, you may contact RTI's Office of Research 
Protection toll free at (866) 214-2043. 
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© Welcome/Definitions 

Definitions: 

Some questions in the survey include a hyperlink (a word or group of words in blue font that is underlined) 
that, when clicked on, brings up a definition. 


For example, 
Career and Technical Education (CTE): The term ‘career and technical education (CTE) means 
organized educational activities that— 

(A) offer a sequence of courses that— 

(i) provides individuals with coherent and rigorous content aligned with challenging academic standards 
and relevant technical knowledge and skills needed to prepare for further education and careers in current 
or emerging professions; 

(ii) provides technical skill proficiency, an industry-recognized credential, a certificate, or an associate 
degree; and 

(ili) may include prerequisite courses (other than a remedial course) that meet the requirements of this 
subparagraph; and 

(B) include competency-based applied learning that contributes to the academic knowledge, higher-order 
reasoning and problem-solving skills, work attitudes, general employability skills, technical skills, and 
occupation-specific skills, and knowledge of all aspects of an industry,including entrepreneurship, of an 
individual. 


Programs of Study (POS): Programmatic coursework that meets the following: incorporates both 
secondary and postsecondary education elements; includes coherent and rigorous content aligned with 
challenging academic standards and relevant career and technical content in a coordinated, non- 
duplicative progression of courses that align secondary education with postsecondary eduction to 
adequately prepare students to succeed in postsecondary education; may include the opportunity for 
secondary education students to participate in dual or concurrent enrollment programs or other ways to 
acquire postsecondary education credits; and leads to an industry-recognized credential or certificate at 
the postsecondary level, or an associate or baccalaureate degree. 


© Welcome/Navigation_Instructions 
Instructions and Navigation: 
NACTE Help Desk: 866-434-4718 


To navigate through the survey you will notice two buttons in the lower left hand corner of each screen titled 
“Previous” and “Next”. You can select either button at anytime to move through the survey. 


Alternately, you can also use the navigation panel located on the left-hand side of the screen. You may jump 
to any question in navigation panel simply by clicking on the question number. 


The questions in the navigation panel will turn green once they have been viewed/answered and 
have passed preliminary validation. 

If a question does not pass preliminary validation either because of incomplete information or 
errors with the response, the question in the navigation panel will turn yellow. 

Questions in the navigation menu that are grey, and have a strike through are not applicable to 
you (e.g., Q62). 


The “Save & Exit” button located in the lower right-hand corner of the screen allows you to exit the survey at 
any time. The next time you login to the survey you will return to the same question. 


Also, located in the lower right-hand corner of the screen is the “Help” button. It contains information 
regarding recommended browsers, and contains definitions for the survey response items (i.e., drop-down 
boxes, checkboxes, and hyperlinks). 


At the end of the survey, please select the “validate” button located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and that there are no errors 
present in your responses. 
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© Programs of Study/Q1 
Did your Local Education Agency (LEA) receive a federal Perkins Title I and/or Title 
II grant in the 2008-09 program year? 


Title II: Tech Prep 


© Programs of Study/Q2 
Did your LEA receive a federal Perkins Title I and/or Title II grant through 
participation in a consortium in the 2008-09 program year? 


Title II: Tech Prep 


© Programs of Study/Q2C_sec 
Where are secondary career technical education (CTE) services delivered in your 
LEA? 


(Mark all that apply) 
I~ Comprehensive high school(s) 


I Regional or area CTE center(s) 
I Full-time technical high school(s) 
I Co-location of programs with a postsecondary institution 


PP Other 


© Programs of Study/Q2D_sec 
In the 2008-09 program year, what was your LEA's total... 


Secondary enrollment (include all students in grades 9-12 regardless of CTE 
Participation) 
Number of CTE participants in Perkins-eligible programs 


Number of CTE concentrators in Perkins-eligible programs 
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© Programs of Study/Q3 


How many Programs of Study (POS) offered in your LEA in the 2008-09 program 
year were developed at the state or local level? 


(Mark "Don't Know” if you do not have this information) 


Locally developed POS, using state template or guidance 


and... 
State approval required and obtained 


© Programs of Study/Q4 


Are any of the POS offered in your LEA described on a Web site? 


© Yes 
© No 


© Programs of Study/Q4_url 
Please provide the web address (url) of a website describing one of the POS offered 


in your LEA. 


What is the URL? | 
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e¢NACTE 
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National Assessment of 
Cereer and Tedinical Education 


© Programs of Study/Q5 

|In the following table, please list the five (5) POS with the highest enrollments in 
| your LEA in the 2008-09 program year and the number of students in each. If you 
have less than five POS then please list them all. FaaSaee Ae REET POS! 


| that did not receive state approval. 


| Type in the name of the POS in the first box, code it into the corresponding Career Cluster area (or 
closest corresponding area) by clicking on the second box, and enter the number of students in the 


third box. Mark ‘Don’t Know’ if you do not have access to this information. 


Number of students 


Name of state-approved POS enrolled in coursework 
in the POS in the 


NOTE: Please crosswalk your POS to the appropriate cluster using the 
pull-down menu provided 2008-2009 program 


TD Don't 


Arts, Audio/Video Technology and Communications 
Business Management and Administration 
Education and Training 

Finance 

|Government and Public Administration 


Health Science 
Hospitality and Tourism 
Human Services 
Information Technology 
||Manufacturing 


Science, Technology, Engineering and Mathematics 
|[Transportation, Distribution and Logistics 
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© Programs of Study/Q6A 


How many of the five POS with the highest enrollments (listed in Q5) incorporated 
each of the following characteristics of a POS? 


Standards 


Select "Don't Know" if you do not have this information 


Aligned with state secondary 
academic standards 


1 1 
pect we sere ester e eee se eee ees ae en ee re ey er ee ae 
' ' 


Aligned with state 
postsecondary standards or 
requirements for program 


c. national based on 16 
career clusters 


Aligned with state college- 
ready standard 


“Aligned with state work-ready : . eee eeu eee pau es eats 
standard te ee ee ee 
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© Programs of Study/Q6B 
How many of the five POS with the highest enrollments (listed in Q5) incorporated 
each of the following characteristics of a POS? 


Curriculum 


‘Select "Don't Know" if you do not have this information 


Uses a curriculum that is... 


Uses a curriculum that spans 
secondary and postsecondary | 


Uses a curriculum that is non- 3 
duplicative across secondary: 
and postsecondary levels 


1 1 1 ' ' 
a ee i 
' ' ' ' ' 


Is part of an articulation 
agreement with ONLY one local 
postsecondary institution 


Is part of an articulation 

agreement with TWO or more : 
local postsecondary institution : 
(in-state or out-of-state) ‘ 


Is covered by a state-wide anes oe ye So rete ee  ..* 
(oe OF O fo a a ee 
articulation agreement ' ' ' ' ' ' : 


“Offers secondary Ctecourses’ = SC=s=‘“‘SSC<‘ SCS 
that earn academic credit 
toward HS diploma 


Offers secondary CTE courses ! 
that earn postsecondary og 
(dual or concurrent credit at 
secondary and postsecondary 
levels) : 
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© Programs of Study/Q6C 
How many of the five POS with the highest enrollments (listed in Q5) incorporated 


each of the following characteristics of a POS? 


Assessments 
Select "Don't Know" if you do not have this information 


Assesses technical skill 


d. state licensing or 
credentialing exams 


(ii) aligned with national 
16 career cluster standards 

(iii) aligned with industry 
standards 

(iv) other 
(specify) 
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© Programs of Study/Q6D_sec 
How many of the five POS with the highest enrollments (listed in Q5) incorporated 


each of the following characteristics of a POS? 


Credentials 
‘Select "Don't Know" if you do not have this information, 


b. that is 
Industry- 
recognized or : 
sponsored 
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© Programs of Study/Q6E 


How many of the five POS with the highest enrollments (listed in Q5) incorporated 
each of the following characteristics of a POS? 


Other Features 


‘Select "Don't Know" if you do not have this information 


guidance 
must be, or ! 
is, available : 


previously 
part of Tech: 
Prep ' 
program 


© Programs of Study/Q7A 
On which of the following topics did your state offer technical assistance to help 
LEA in your state in developing POS for state-approval? 


(Mark all that apply) 
TF POS templates or guidelines 


P Career clusters 

M CTE content standards 

TM CTE curriculum development guidelines 

[ Academic and CTE curriculum integration 

I Secondary and postsecondary curriculum alignment 
PM Technical skill assessments 

I~ Career guidance and counseling 

I~ Data systems for monitoring student progress 

I Aligning standards and assessments 


[— Other 
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© Programs of Study/Q7B 
In which of the following areas (identified in Question 7A) did your LEA staff 
participate during the 2007-08 or 2008-09 program years? 


(Mark all that apply) 
I~ POS templates or guidelines 


DP Career clusters 

I CTE content standards 

lM CTE curriculum development guidelines 

[7 Academic and CTE curriculum integration 

I~ Secondary and postsecondary curriculum alignment 
PF Technical skill assessments 

I~ Career guidance and counseling 

[ Data systems for monitoring student progress 


F Aligning standards and assessments 


© Programs of Study/Q8 

What types of professional development does your state offer AND in what types of 
activities did your LEA staff participate in to support the development and 
implementation of POS? 


ee all that apply for each group. If your state offered other types of professional development, 
the f professional development in textbox(es) in the "Other" column below. 


Local 
workshops workshop/ 
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© Programs of Study/Q9A_sec 
To what extent did each of the following stakeholder groups participate in your 
LEA's efforts to develop POS? 


Local chamber of commerce 
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© Programs of Study/Q9B_sec 
To what extent did each of the following stakeholder groups participate in your 
LEA's efforts to implement POS? 


Local chamber of commerce 
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© Programs of Study/Q10A 
To what extent has each of the following barriers limited your LEA's efforts to 
develop POS? 


Lack of cooperation by secondary academic ! 


‘ ' ' ' 
ae eee ee eee eee, ee ee ee 
1 1 ' 


Lack of cooperation by secondary 


Lack of cooperation from postsecondary 
faculty or administrators 


' ' ' ' 
at a a ee 120 ene eee eeeeeenneeeeseen eee eee eee 
1 1 1 1 


Lack of cooperation by state level staff 


Lack of state leadership 
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© Programs of Study/Q10B 
To what extent has each of the following barriers limited your LEA's efforts to 
POS? 


Lack of cooperation by secondary academic | 
faculty 


' ' ' 
Se oe 
' 1 1 


Lack of cooperation by secondary 


Lack of cooperation from postsecondary 
faculty or administrators 


Lack of state leadership 
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© Programs of Study/Q10C_sec 


Please indicate the earliest grade level at which each of the following types of 
career guidance is offered to students participating in state-approved POS in your 
LEA. 


Career days (with guest speakers) 

Career fairs (with local business represented) 
Job shadowing 

Internships/Co-ops 


Counselor meetings with individual students and parents to discuss career 
preparation and course selection 


Student development of a written plan of courses (e.g., an individual 
graduation plan) 


Student written selection of a POS 


Student completion of a career interest assessment 


© Programs of Study/Q10D_sec 


Please indicate whether each of the following types of career guidance activities is 
of students at any point in their program. 


Counselor meetings with individual students and parents to discuss : 
career preparation and course selection : 


Student development of a written plan of courses (e.g., an individual 
graduation plan) 


Student completion of a career interest assessment 
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© Accountability/ACC_INTRO 
The questions in the remainder of this survey address all of the CTE _ 
programs offered in your LEA irrespective of whether or not they qualify as a 


© Accountability/Q11_sec 
How difficult was it for your LEA to reach agreement with your state agency on 
performance benchmarks and targets for each of your Perkins accountability 


Not Somewhat Very 
Difficult Difficult Difficult 
ee 


measures? 


Academic 
attainment (1S1, 


Technical skill 
attainment (2S1) 


completion (3S1) ; 


1 1 1 
that att tt tn ttt tt ti al 
' 1 ' ' 


Student 
graduation rates 


completion (6S2) 


© Accountability/Q11B 
When negotiating with your state education agency to establish annual performance 
benchmarks and targets for your LEA, what were the reasons for encountering 


difficulty? 


(Mark all that apply) 
I Insufficient baseline data upon which to negotiate targets 


I Lack of clear guidance from the state on what constitutes continuous improvement 
I Anticipated difficulty in gathering and submitting required data 

Il Anticipated difficulty in being able to meet proposed performance targets 

[™ Concern over the repercussions of your LEA failing to meet performance targets 


Other 
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© Accountability/Q12 
How do you identify students meeting the CTE concentrator threshold for Perkins 
accountability purposes? 


(Mark all that apply) 
I~ Students self-report concentrator status 


I~ Teachers identify students based on course taking 

[ Administrators identify students using local management information system 
State identifies students using statewide database; no action at LEA level 

[7 Other 


I Don't know 


© Accountability/Q13 
How do you ensure that the accountability data you collect for the Perkins 
accountability measures are valid and reliable? 


(Mark all that apply) 
I Follow state guidance on measure development 


lM Provide technical assistance to teachers 

[™ Require teachers or administrators to review or spot-check data submissions 
[ Administrators or program chairs conduct an internal review of data 

I~ Use a software program that automatically flags errors 

[> Other 

[ No steps taken to ensure data accuracy 


I~ Don’t know 
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© Accountability/Q14_sec 
How confident are you that the data you are collecting for each of the following 
Perkins core indicators of performance provides an accurate measure of your LEA's 


actual performance? 


CTE Populations 


Not mewhat Very Don't 
Confident paresis Confident Know 


CTE participants ; 


Core Indicator 3 


Not ewhat Very 
Confident peat tlk Confident 


credential/ certificate/ —, 


Continued on next page... 
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© Accountability/Q14_sec 
How confident are you that the data you are collecting for each of the following 
Perkins core indicators of performance provides an accurate measure of your LEA's 


actual performance? 


Continued.... 


Core Indicator 4 


Not Somewhat Very 
Confident Confident Confident 


Non-traditional 
completion 
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© Accountability/Q15 


How confident are you that the data you are collecting for each of the Perkins 


special population categories provides an accurate measure of the actual 


performance of these populations? 


Not Somewhat 
Confident Confident 


Individuals from 
economically 
disadvantaged 
families, including 
foster children 


Single parents, 
including single 
pregnant women 


Individuals with 
limited English 
proficiency 


Migrant students 


© Accountability/Q164 
In what ways and to what extent did your LEA use its prior year Perkins 
accountability data in the 2008-09 program year? 


Used not | Used not 
at all very much | somewhat 


To identify programs in need 


To identify special population 
students not being aioe a 


Overall, how useful are the 
Perkins data you collect for 
your local program 
improvement efforts? 


Local Education Agency (LEA) Survey 


Page 21 of 27 


e¢NACTE 


National Assessment of 
Cereer and Technical Eduestion 


© Accountability/Q16B 

In the previous question you indicated that your LEA did not use its prior year 
Perkins accountability data in the following ways in the 2008-09 program year. Are 
there plans to use the data for any of these purposes in the next two years? 


To provide targeted technical assistance 
‘To identify special population students not being | ll 
adequately served ' ' ' 


‘ ‘ ' 
ainda ammeter uma aa ea h  ree ineein RT a eS 
‘ ' 


Other c c : c 


© Accountability/Q17A 


What type of feedback does your state agency provide you on your LEA's 
performance on the Perkins accountability measures? 


(Mark all that apply) 
I No data on my LEA performance is shared 


[7 Statewide average performance outcomes for all LEAs are shared 
[7 Performance outcomes are shared for LEAs with characteristics similar to my own 


I~ Performance outcomes are shared for all individual LEAs 


© Accountability/Q17B 

If your LEA were to fail to meet its Perkins performance targets, does your state 
have a policy for the type of corrective action you must take to address the 
accountability deficiencies? 


(Mark all that apply) 
I~ Not applicable, no state policy exists 


I~ Develop a program improvement plan 

I Provide professional development to staff of underperforming programs 
I Consult with state or local business or industry experts in the field 

I Eliminate underperforming programs 

I Restrict student access to programs 

I Restructure or redesign existing CTE programs 

D Other (specify) 


[ Don’t know 
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© Accountability/Q18 

What proportion of the students participating in state-approved CTE programs 
within your LEA were assessed using one of the following methods to measure their 
technical skill attainment in the 2008-09 program year? 


i? 
None Some eau 


State-developed secondary 


National licensing or 
credentialing exams 


Grade point average (GPA) 


Course or program completion | 


© Accountability/Q20 

The 2006 Perkins Act introduced a new set of accountability requirements for Tech 
Prep students. Indicate your level of agreement or disagreement with each of the 
following statements: 


New measures impose a 
substantial data burden 


The new measures will support: 
program improvement efforts : 
at the 


The new measures will support: 
program improvement efforts : 


Data collected for the 
measures are of high quality 


The advantages of the new 
measures outweigh their 


My LEA is able to collect data 
on all measures 


obtaining state guidance on 
collecting data for the Tech 
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© Finance/Q21 


How much of the total budget for CTE offered in your LEA in the 2008-09 program 
year came from each of the following sources? 


If none, enter “0”, Mark “Don’t Know” if you do not have this information 


Daler amount | Ren 


State funds: 
Formula (must be spent on CTE) 


Other federal funds (specify) | 
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© Finance/Q22 


For which of the following permissible uses referenced in Section 135(c) of the 
2006 Perkins Act did you use your 2008-09 Title I basic grant funds? 


(Mark all that apply) 

IT Involving business and labor in designing, implementing, and evaluating CTE programs 

covered by the Act 

I Providing career guidance and academic counseling 

I~ Promoting work-related experiences for students 

I~ Promoting industry experiences for teachers 

I Providing programs for special populations 

Il Assisting CTE student organizations 

I~ Offering mentoring and related support services 

I Leasing, purchasing, upgrading, or adapting equipment 
[™ Supporting teacher preparation programs 
Entrepreneurship education and training 

[ Developing new CTE courses 

[7 Offering continuing education or job referral services 

[ Supporting nontraditional training and activities 

Il Providing training programs in automotive technologies 
IT Improving accountability data collection and reporting 


[~ Implementing CTE programs of study 


[7 Other 
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© Finance/Q23 


What impact has each of the following changes introduced in the 2006 Perkins Act 
had on the administration and implementation of CTE programs in your LEA? 


Somewhat Somewhat 
Negative Positive 
Impact Impact 


Very Positive 
Impact 


Requirement to 
offer atleastone  : 
POS to obtain funds : 


Requirement to 
dedicate funds to 
support special 
populations in 
preparing for high 
wage/skill/demand 


of funds to support 
auto technologies: 


New allowable use 
of funds to form 
consortia to 
maximize 
investment and 
encourage 
innovation 


Requirement to : 
report 
disaggregated 
performance 
outcomes for 
populations 
referenced in the 
Elementary and 
Secondary 

Education Act : 
(ESEA), also known : 
as the No Child Left : 
Behind Act : 


Requirement to 
negotiate 

performance 

benchmarks with =: 

the State agency to : 

qualify for funds 


saneannnaaeaeneennneenes RN Peer 
Sanctions for failing : : 

to meet state ‘ ' : ' : 

negotiated : Cc c ‘ c c 2 
performance : : : ‘ : 

benchmarks 
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© Finance/Q24_sec 

If your LEA was unable to enter into a consortium and received a state waiver for 
the minimum allocation rule ($15,000 per substate grantee) in 2008-09, why 
was it granted? (Reference: Section 131 (c) (1) and (2)] 


© Not applicable, no waiver granted 
© Rural, sparsely populated area 

© public charter school 

© other (specify) 

© Don't know 


© END 
Please located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and 
that there are no errors present in your responses. If there are errors present, 
simply select the question from the validation summary list to take you directly to 
the question to correct your data. Once you have corrected the data, go to the 
navigation panel on the left side of the screen and select the form titled END. 


You have completed the Survey of Secondary School Districts Offering Career- 
Technical Education. 


Thank you very much for taking the time to participate in this important study. 


If you have any questions or comments about this study, please contact Steven 
Klein (MPR) at 503-963-3757 
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National Assessment of 
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© Welcome/Contact_info 


Welcome to the Web-based National Assessment of Career and Technical 
Education survey (NACTE) sponsored by the U.S. Department of Education. 


Please take a moment and complete the contact information below before 
proceeding with the survey. 


First Name: _ '& 
Last Name: | 
Title: | 
Phone (XXX-XXX-XXXX):[.—<“Ci‘CS™S™S™SC~*™S 
Email: l.. 
Address: 

Address 2 (optional): 

City: 

State: 


Zip code: 


© Welcome/Confidentiality 


Uses of the Data: 

This study is being conducted for the U.S. Department of Education, which has contracted 
with MPR Associates, Inc., and RTI International (RTI). The purpose of this study is to collect 
information relating to the implementation of the Carl D. Perkins Career and Technical 
Education Act of 2006 during the 2008-09 academic year. Because a limited number of 
institutions have been selected, we would very much like for you to participate on behalf of 
your educational institution. 


Confidentiality 

As a matter of policy, The U.S. Department of Education is concerned with protecting the 
privacy of participants in voluntary surveys. 

We want to let you know that: 


Your responses will be merged with those of other respondents and will not be 
identified as the agency you represent, except as required by law. 

You may skip questions you do not wish to answer; however, we encourage you 
to answer as many questions as possible, because incomplete data will reduce the 
value of the information provided to Congress. 

Your answers may be used only for statistical purposes. Participation is voluntary 
and will not affect any aid or other benefits that your state agency, LEA, or 
institution may receive. 


If you have questions about the study, you may contact: 


the NACTE Help Desk at (866) 434-4718 

Steven Klein, the study's director, at (503) 963-3757 or sklein@mprinc.com, or 
Michael Fong, the U.S. Department of Education's Project Officer, at (202) 401-7462, 
Michael. Fong@ed.gov. 


If you have questions about your rights as a study participant, you may contact RTI's Office of Research 
Protection toll free at (866) 214-2043. 
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Some questions in the survey include a hyperlink (a word or group of words in blue font that is underlined) 
|that, when clicked on, brings up a definition. 


|For example, 
Career and Technical Education (CTE): The term ‘career and technical education (CTE) means organized 
jeducational activities that— 
(A) offer a sequence of courses that— 
(i) provides individuals with coherent and rigorous content aligned with challenging academic standards 
jand relevant technical knowledge and skills needed to prepare for further education and careers in current or 
emerging professions: 
(ii) provides technical skill proficiency, an industry-recognized credential, a certificate, or an associate 
jdegree; and 
(ili) may include prerequisite courses (other than a remedial course) that meet the requirements of this 
subparagraph; and 
(B) include competency-based applied learning that contributes to the academic knowledge, higher-order 
reasoning and problem-solving skills, work attitudes, general employability skills, technical skills, and 
occupation-specific skills, and knowledge of all aspects of an industry.including entrepreneurship, of an 
individual. 


Programs of Study (POS): Programmatic coursework that meets the following: incorporates both 
secondary and postsecondary education elements; includes coherent and rigorous content aligned with 
|challenging academic standards and relevant career and technical content in a coordinated, non-duplicative 
| progression of courses that align secondary education with postsecondary eduction to adequately prepare 
|students to succeed in postsecondary education; may include the opportunity for secondary education 
students to participate in dual or concurrent enrollment programs or other ways to acquire postsecondary 
jeducation credits; and leads to an industry-recognized credential or certificate at the postsecondary level, or 
an associate or baccalaureate degree. 


© Welcome/Navigation_Instructions 
Instructions and Navigation: 
NACTE Help Desk: 866-434-4718 


To navigate through the survey you will notice two buttons in the lower left hand corner of each screen titled 
“Previous” and “Next”. You can select either button at anytime to move through the survey. 


Alternately, you can also use the navigation panel located on the left-hand side of the screen. You may jump 
to any question in navigation panel simply by clicking on the question number. 


The questions in the navigation panel will turn green once they have been viewed/answered and 
have passed preliminary validation. 

If a question does not pass preliminary validation either because of incomplete information or 
errors with the response, the question in the navigation panel will turn yellow. 

Questions in the navigation menu that are grey, and have a strike through are not applicable to 
you (e.g., 62). 


The “Save & Exit” button located in the lower right-hand corner of the screen allows you to exit the survey at 
any time. The next time you login to the survey you will return to the same question. 


Also, located in the lower right-hand corner of the screen is the “Help” button. It contains information 
regarding recommended browsers, and contains definitions for the survey response items (i-e., drop-down 
boxes, checkboxes, and hyperlinks). 


At the end of the survey, please select the “validate” button located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and that there are no errors 
present in your responses. 
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© Programs of Study/Intro 


ques 


© Programs of Study/Q1 
Has your state passed legislation supporting the development and/or 


implementation of Programs of Study (POS) since the reauthorization of the Carl D. 
Perkins Career and Technical Education Improvement Act of 2006 (Perkins)? 


© Yes, in response to the 2006 Perkins reauthorization 

© Yes, for reasons other than the 2006 Perkins reauthorization 
© No, already have supporting legislation in place 

[No 

© Don't know 


© Programs of Study/Q1_upload 


Please attach a copy of the legislation supporting the development and/or 
implementation of POS or provide web link. 


Provide your file in ane of these formats: .doc, .xis, .csv, .dat, .txt, .pdf. If you 
have sore other file type, please call our help desk at 866-434-4718. 


Please attach a copy of the legislation or provide a web link: 


T™ Do not have electronic copy of legislation 


© Programs of Study/Q2 
Has your state modified or expanded its administrative policies supporting POS 


development and/or implementation since the reauthorization of the 2006 Perkins 
Act? 


© Yes, in response to the 2006 Perkins reauthorization 

Yes, for reasons other than the 2006 Perkins reauthorization 
No, already have supporting legislation in place 

[No 

© Don’t know 
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© Programs of Study/Q2_upload 


Please attach a copy of the policies supporting POS development and/or 
implementation or provide web link 


Provide your file in one of these formats: .doc, .x!s, .csv, .dat, .txt, .pdf. If you 
have some other file type, please call our help desk at 866-434-4718. 


Please attach a copy of the policies or provide a web link: 


T Do not have electronic copy of legislation 


© Programs of Study/Q3 
Is state approval required for all POS offered within postsecondary institutions? 


© Yes, state approval is required for all POS 

No, state approval is only required for a subset of POS 

© No, state approval is required only for the one (1) POS required of each local agency 
© No, state approval is not required 

© Don’t know 


© Programs of Study/Q4 
Does state administrative policy require that all career and technical education (CTE) 
programs in your state eventually become POS? 


© Yes, during the lifetime of the Perkins Act of 2006 
© Yes, no timeline specified 
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© Programs of Study/Q5 


How many state-approved POS offered in your state in the 2008-09 program year 
were developed either at the state agency or institutional level? 


Enter number (whole numbers only, do not include commas or other non-numeric 
characters), mark "Don't Know” if you do not have this information 


State developed POS 
State- approved, locally developed POS, using state template 
or guidance ; 


State-approved, locally developed POS (i.e., no state guidance 
provided to institutions) : 


Total : ' 


© Programs of Study/Q6 


What percentage of postsecondary students participating in CTE in your state were 
enrolled in coursework that was part of a state-approved POS in the 2008-09 
program year? 


© None 

© Less than 25% 

© Between 26 to 50% 
© Between 51 to 75% 
© Between 76 to 99% 
© all 

© Don’t know 
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© Programs of Study/Q7 


In the following table, please list the five (5) POS with the highest enrollments in 
your state in the 2008-09 program year and the number of students enrolling in 
each. If you have less than five POS then please list them all. Please do not include 
any POS that did not receive state approval. 


Type in the name of the POS (or name that best describes the POS offerings at the 
local level) in the first box, code it into the corresponding Career Cluster area (or 
closest corresponding area) by clicking on the second box, and enter the number of 
students enrolling in the POS in the third box. Mark ‘Don’t Know’ if you do not have 
access to this information. 


Number of students 


Name of state-approved POS enrolled in coursework 
NOTE: Please crosswalk your POS to the appropriate cluster using in the POS in the 
the pull-down menu provided 2008-2009 program 
year 


1 , 
: : --select one—- hd | —_ | on 
5 ' |=eelect one— >| | [ : ysl 
? aa ; Cl ee 
° : —select one—- hd [ rent 
7 | rr [Don't 


Know 


Agricultural, Food, and Natural Resources 
Architecture and Construction 

Arts, Audio/Video Technology and Communications 
Business Management and Administration 
Education and Training 

Finance 

Government and Public Administration 


Health Science 
Hospitality and Tourism 
Human Services 


Information Technology 

Manufacturing 

Marketing 

Public Safety 
Science, Technology, Engineering and Mathematics 
Transportation, Distribution and Logistics 
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© Programs of Study/Q8A 

hich of the following elements of a POS does your state require institutions to 
adopt as part of a state-approved POS? (Note: These elements are not specifically 
defined within the Perkins legislation, but may be required by states as part of a 


locally developed POS.) 


Standards 


b. industry- developed 
c. national based on 16 career clusters 


© Programs of Study/Q8B 
Which of the following elements of a POS does your state require institutions to 
adopt as part of a state-approved POS? (Note: These elements are not specifically 


defined within the Perkins legislation, but may be required by states as part of a 
locally developed POS.) 


Curriculum 


Uses a curriculum that spans the secondary and 
postsecondary levels 


Offers postsecondary credit to secondary students 
participating in dual or concurrent enrollment programs : 
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© Programs of Study/Q8C 
Which of the following elements of a POS does your state require institutions to 


adopt as part of a state-approved POS? (Note: These elements are not specifically 
defined within the Perkins legislation, but may be required by states as part of a 


locally developed POS.) 
Assessments 


Assesses technical skill attainment through... 


a. state-developed exams that are... 
(i) aligned with state technical standards 


(ii) aligned with national 16 career cluster 
standards 


e. locally developed exams that are... 
(i) aligned with state standards 


(ii) aligned with national 16 career cluster 


(iv) aligned with other 
(specify) 
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© Programs of Study/Q8D_postsec 

Which of the following elements of a POS does your state require institutions to 
adopt as part of a state-approved POS? (Note: These elements are not specifically 
defined within the Perkins legislation, but may be required by states as part of a 
locally developed POS.) 


Other features 


Respond to local high skill, high demand, high pay 
occupational area 


Career guidance must be available 


© Programs of Study/Q9 
What action has your state taken to ensure that POS development at the 
institutional level aligns with the core elements identified in the Perkins legislation? 


Mark all that apply 

T Creating POS models or templates 

T™ Developing curriculum guides and other materials 

T Designing guides for aligning CTE content with academic standards or learning outcomes 
T Designing guides for aligning CTE content with technical standards 

T™ Requiring institutions to provide evidence of alignment in their local plans 

T™ Securing secondary/postsecondary agreement on specific POS elements 

T™ Consulting the career clusters developed by the States’ Career Clusters Initiative 


T Providing training and professional development to institutional staff 
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© Programs of Study/Q10A_postsec 
To what extent did representatives from the following stakeholder groups 
participate in state agency efforts to develop state-approved POS? 


> 
oT 
cr 


Postsecondary academic faculty 
Postsecondary CTE faculty 


O mo 


- + Peaeenad----- 


National industry/union groups or 
professional associations 


Local chamber of commerce 


' 
' 
' 
1 
¥ 
' 
. 
' 
' 
- 
' 
' 
' 
' 


© Programs of Study/Q10B_postsec 


Were representatives of the following stakeholder groups required to participate or 
had the option to participate in institutional efforts to develop state-approved POS? 


Local chamber of commerce 
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© Programs of Study/Q11 

On which of the following topics did your state introduce new technical assistance 
activities to help institutions develop state-approved POS during the 2007-08 
and/or 2008-09 program years? 


Mark all that apply 

[Pos templates or guidelines 

[Career clusters 

[CTE content standards 

[CTE curriculum development guidelines 
[Academic and CTE curriculum integration 
Secondary and postsecondary curriculum alignment 
[Technical skill assessments 

[Career guidance and counseling 

[Data systems for monitoring student progress 
[aligning standards and assessments 

[Other 
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© Programs of Study/Q12_postsec 

What types of professional development did your state offer to the following groups 
involved in the development and implementation of state-approved POS during the 
2007-08 and /or 2008-09 program years? 


Mark all that apply for each group. If your state offered other types of professional 
development, specify the type(s) of professional development in textbox(es) in the 
"Other" column below. 


Postsecondary : 
administrators : 


Postsecondary | 
faculty : : : 


Secondary 
administrators : 


Secondary 


counselors! 


Secondary ' 
CTE teachers | 
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© Programs of Study/Q13 


Indicate whether you agree or disagree with the following statements that describe 
your state agency’s efforts to develop state-approved POS. 


Strongly Strongly | Does not 
ee ee ee 


A shortage of 
state staff has 


a es 


State staff have 
technical 
expertise in 
designing POS 


spr cc rrr ee eee 


There is adequate ! 
federal guidance 
to support POS 
development 


Lack of statewide | 
technical content ' 
standards has 
hindered POS 
development 


Lack of statewide : 


Lack of technical 
skill assessments | 
has hindered POS : 
development 


State staff have 


little influence at | 
the local level 


--CONTINUED ON NEXT PAGE-- 


--Continued from previous pag 
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© Programs of Study/Q13 
Indicate whether you agree or disagree with the following statements that describe 
your state agency’s efforts to develop state-approved POS. 


aie Strongly | Does not 
ee Agree Apply | 


Postsecondary ; 
CTE faculty have | 
a good 
understanding of ; 


SS oo 


ee 


Secondary CTE 
teachers and 
postsecondary 

CTE instructors 
lack the ' 
necessary skills or 
time to integrate 
academic and 
technical 
standards to 


design curricula 


oo + fea = se aw ea snes saa Sa eae} --------------------- 


-- > {esse eae ease = sea aa aes} ------- 
oo + fea a ea aw aa eae ae naan aaa} --------- - 
fen ece ne ee scene eee ne ede 


Secondary 
teachers and 
postsecondary 
instructors are 
cooperating to 
create POS 


Secondary and 
postsecondary 
state staff are 
cooperating to 
implement POS 


wn +--+ pean --------------- 
es 
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© Programs of Study/Q14 
How are CTE content standards developed for the state-approved POS offered in 
your state? 


Mark all that apply 

[State develops postsecondary CTE standards 

[State adopts industry-based standards 

[State adopts 16 career clusters standards 

[Local instructors develop content standards based on state criteria 

[Local instructors develop their own standards 

[Local instructors adopt standards created by industry groups 

[Local instructors consult with business or advisory council to develop or select standards 
I No standards exist 


Don't know 


© Programs of Study/Q15 
What proportion of the state-approved POS offered in your state used each of the 
following sources for CTE content standards in the 2008-09 program year? 


Locally developed CTE 
standards 


National standards for the 16 
career clusters 


No CTE standards are 
required or exist 


aon - e+ fewn oneal ----- 
------ ee -- 
------ Seeeieh -- ---- 
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© Programs of Study/Q16 
What share of POS approved by your state incorporated each of the following levels 
of CTE content standards in the 2008-09 program year? 


Cluster level skills (i. e., 16 
national career clusters) 


Pathway level skills (i.e., 79 
national pathways) - 


Occupational level skills (e. g-, | 
job-specific) ‘ 


© Programs of Study/Q17 
How many of the state-approved POS offered by institutions in your state used the 
following tools to assess students’ technical skill attainment in the 2008-09 program 


Locally-developed CTE skill 
exams 


Industry developed, ‘employer ee ae ee 
validated exams 


National licensing or 
credentialing exams 


Commercially- developed 
exams (e.g., NOCTI) 


Grade point average (GPA) 


Course or program comeleliia. 
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© Programs of Study/Q19A 


How are institutions monitored by the state to ensure they are offering state- 
? 


Mark all that apply 

[State agency staff visit individual sites 

T™ Secondary LEA administrators of CTE visit individual sites 
[Through state reporting requirements for POS 

T Other approach 


[Not applicable/Not done 


© Programs of Study/Q19B 


How are POS offered within institutions evaluated to assess a 
? 


Mark all that apply 
[State agency staff visit individual sites 


l™ Secondary district office staff visit individual sites 
[Through state reporting requirements for POS 
[Other approach 

[Not applicable/Not done 
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© Programs of Study/Q20A 
Who is responsible for monitoring the implementation of state-approved POS for 
Perkins compliance? 


Mark all that apply 

Il” State agency staff 

[LEA administrators of CTE 

l™ Secondary teachers or administrators 
T™ Postsecondary faculty or administrators 
T Advisory committees 

T™ Other agency or individual 


[Not applicable/Not done 


© Programs of Study/Q20B 


What changes or additions to the Perkins legislation regarding POS would you 
recommend that Congress consider for the next reauthorization? 


—— 
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© Accountability/AccountabilityIntro 


The questions sip apron a uacbaucatvidigsl Got dat aeeeee mor eet 
ective of whether or not they qualify 


© Accountability/Q21 
Overall, how important were the following issues in negotiating your state 
performance benchmarks and targets with the U.S. Department of Education? 


Past state H 
performance on Perkins | 
measures 


Anticipated changes in 
CTE program offerings : 


-- - {eee ------ Coen 


‘Anticipated change in in + 
state and/or local 
funding 


Other 
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© Accountability/Q22 

How difficult was it to reach agreement when negotiating with the U.S. Department 
of Education to establish annual performance benchmarks and targets for your 
state? 


Annual FAUPL 
performance 
targets 


© Accountability/Q23 
When negotiating with the U.S. Department of Education to establish annual 
performance benchmarks and targets for your state what were the reasons for 


encountering difficulty? 

Mark all that apply 

[Insufficient baseline data upon which to negotiate targets 

[Lack of clear guidance from the Department on what constitutes continuous improvement 
C anticipated difficulty in gathering and submitting required data 

T anticipated difficulty in negotiating similar performance targets with local grantees 


[Concern over the repercussions of the state failing to meet performance targets 


T other 


© Accountability/Q24 


What percentage of institutions in your state have negotiated performance 
improvement targets differing from your state-established targets on one or more 
performance measures in 2008-09? 


Enter percentage (whole numbers only, do not include % or other non-numeric 
characters) or mark “Don’t Know” if you do not have this information 


Performance levels and improvement targets are individually ; Cl 
negotiated for what percentage of institutions? ‘ 
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© Accountability/Q25 
How important were the following factors when negotiating performance targets 
with institutions? 


Past performance on 
Perkins measures 


Anticipated changes in 
CTE program offerings : 


Anticipated changes in 
state and/or local 
funding 


Other 


— 
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© Accountability/Q26_postsec 

OVAE issued non-regulatory guidance to assist states in developing their 
accountability systems for the 2006 Perkins Act. How did your state make use of 
the OVAE guidance in crafting each of the following? 


Identifying Populations 


CTE participants 


CTE concentrators 


Attainment of an industry-recognized 
credential certificate, or degree (2P1) 


Placement into an apprenticeship 
program, the military, or employment 


Non-traditional completion (5P2) 
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© Accountability/Q27 


Which of the following types of technical assistance did your state provide to assist 
institutions in implementing accountability measures and reporting during the 
2008-2009 program year? 


Mark all that apply 

[Issued guidance on data collection 

T™ Statewide training on data collection and reporting 
T Local site visits 

[Help line (telephone or email) 

T Provided individualized technical assistance 

T Other 


[Did not provide any technical assistance 


© Accountability/Q28 
Does the database you use for Perkins reporting purposes enable you to: 


Monitor individual students over time 


Access non-CTE-related information contained in 
student transcripts 


Link high school records to administrative records for 
students enrolling in in-state community colleges 


Link high school records to administrative records for | 
students enrolling in in-state four-year colleges or 
universities 


Link high schools records to administrative records for : 
students enrolling in out-of-state community colleges, : 
four-year colleges, or universities ' 


ee ee) - 
onan en sne} 


Link in-state community college records to in-state 
4-year colleges or universities 
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© Accountability/Q29 


What types of quality control measures has your state adopted to ensure the 
validity and reliability of institutionally reported data? 


Mark all that apply 
[ Conduct desk audit of local data to identify inaccurate information 
[Use electronic error checking strategies to identify inaccurate data 


[Provides technical assistance and/or guidelines to locals on data collection and editing 
procedures 
T™ State compares totals to prior year reports 


Tl other 


[No quality control measures have been adopted 
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© Accountability/Q30_postsec 

How confident are you that the data you are collecting from institutions for each of 
the following Perkins core indicators of performance accurately reflect local 
performance? 


Core Indicator 1 


Core Indicator 2 


Transfer to a 
baccalaureate 
degree 
program 


--Continued on the next page-- 
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© Accountability/Q30_postsec 
How confident are you that the data you are collecting from institutions for each of 
the following Perkins core indicators of performance accurately reflect local 


performance? 


Core Indicator 4 


Placement in 


apprenticeship 


Non-traditional 
completion 
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© Accountability/Q31 


How confident are you that LEAs can report -- complete and accurate data for the 
following special populations, as defined by ESEA? 


Not Somewhat Very Don’t Not 
Confident Confident Confident | Know Applicable 


Individuals ‘ ' ' 
with 
disabilities 


Individuals : ‘ : 

from ‘ i 

economically 
disadvantaged: 
families, \ 
including 
foster children: 


Individuals 
preparing for 
nontraditional 
fields 


oo 


' 

' 

' 

' 

' 

' 

' 

' 

' 

' 

' 
' ' 1 
nm ar a 
1 ' ' 

. 

1 ' ' 
1 ' 1: 
. 

1 ' ' 
1 1 1 
1 ' ' 
1 ' ' 
1 ' 1 
1 ' ' 
1 


parents, 
including 

single 

pregnant 
women 
Displaced ; 
homemakers 


Individuals 
with limited 
English 
proficiency 
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© Accountability/Q32_postsec 
What proportion of institutions in your state were unable to achieve their negotiatec 
performance target for each of the following Perkins accountability measures during 


the 2008-09 program year? 


Attainment of an 
industry- 
recognized 
credential 
certificate, or 
degree (2P1) 


Placement into an : 
apprenticeship 
program, the 
military, or 
employment (4P1) : 


Non-traditional: 
completion (5P2) 


© Accountability/Q34 

How many institutions in your state are required to develop a Perkins program 
improvement plan for 2009-10 on the basis of performance data that were 
generated in the 2008-09 program year? 


Enter number (whale numbers only, do not include commas or other non-numeric 
characters), mark “Don't Know" if you do not have this information 


Institutions in need of improvement 
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© Accountability/Q354 


In what ways and to what extent does your state currently use Perkins 
postsecondary core indicator data? 


Used toa 
great 
extent 


Used quite Used Used not | Used not at 
a bit somewhat | very much all 


To reward institutions 
that are performing well : 


To sanction 
low-performing 
institutions 


To identify programs in 
need of improvement 


To provide targeted 
technical assistance 


To provide additional 
resources for program 
improvement 


b---------4------ 


To identify unusually 
effective programs 
To identify special 
populations not being 
adequately served 
Other 


(specity) 


' 
-- (eee eeewem ------- 
« 


NACTE State Director Survey — Postsecondary 29 of 51 


ae Reference copy 
ey NACTE Please complete 
Pastel cache tee rk oer 


© Accountability/Q35B 

On the previous question you indicated your institution did not use its prior year 
Perkins accountability data in 2008-09 in the following ways. Are there plans to use 
the data for these purposes in the next two years? 


To reward institutions that are performing well 


To sanction low-performing institutions 


To provide additional resources for program 
i t 


To identify unusually effective programs 
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Other (if specified) 


© Accountability/Q36 


What types of consequences do existing state policies impose on institutions that do 
not meet one or more of their Perkins I'v accountability performance targets? 


Mark all that apply 

T Consequences exist but are unlikely to be imposed 

T Develop a local program improvement plan 

T Additional accountability reporting requirements 

Loss of, or decrease in, state or Perkins funding 
[Restrictions or loss of flexibility in the use of Perkins funds 
T other 

[No consequences exist 


TM pDon't know 
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© Accountability/Q37 
How far along is your state in the development of technical skills assessments? 


Mark all that apply 

[created state-developed technical skills assessment Covering al! CTE programs 
[created state-developed technical skills assessment Covering a subset of CTE programs 
T adopted commercial technical skills assessment Covering al! CTE programs 

T adopted commercial technical skills assessment Covering a subset of CTE programs 

T Approved list of technical skills assessments for local use 

[In process of designing or transitioning to CTE technical skills assessment system 

T Considering development options 

T other 


[No plans to develop an assessment system at this time 
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© Accountability/Q38 


How many of the institutions in your state used the following tools to assess 
students’ technical skill attainment in the 2008-09 program year? 


State developed 
postsecondary CTE skill 
exam(s) 


Institutionally developed CTE : 
skill exam(s) 


Industry developed, nS Se 
validated exam(s) © 


National licensing or 
credentialing exar(s) 


State licensing or 
credentialing exam(s) 


Commercially developed 
exam(s) (e.g., NOCTI) 


fo-----------f------4 


CTE course or program 
completion 
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© Accountability/Q39 


What percentage of CTE concentrators reported in your state’s technical skill 
assessment measure were assessed using a state or industry developed technical 
skill assessment, national or state licensing or credentialing, or commercially 
developed exam in the 2008-09 program year? 


Less than) Between | Between | Between All Don’t 
25% 26 to 5096| 51 to 7596) 76 to 99% know 


State ' ‘ 
developed 
technical 
skill 
assessment | 


1 ' 1 
. 

' ' 1 

' ' 1 

' ' 1 

' ' 1 

' ' 1 

' ' 1 
. 

' ' 1 
. 

! ' 1 

+ + - 
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Industry 
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technical 

skill 
assessment |: 
National 
licensing or 
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licensing or 
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exam i 
Commercially : 
developed 
exam 


e 
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© Accountability/Q40 
For each of the following data sources, indicate whether your state uses this source 
to collect placement data and, if not in use, whether state legal restrictions prevent 


your state in using this approach? 


State Unemployment Insurance 
wage records 


State higher education student 
database 
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© Accountability/Q41 


How does each of the following impact your state’s capacity to report placement 
data? 


Cost of 
conducting : 
follow-up studies ; 
of students who 
have left high 


~~ -pa====aaan}---------- o-- -- 
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Collecting data on; 
those employed: 
out-of-state 


Collecting data on 
those attending 
an in-state 
4-year public 
postsecondary 
institution 


See eee nes 


a 
Sed tty as weece 
2 ee 


Collecting data on 
those attending 
an out-of-state 
public or private 
postsecondary 
institution 
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© Accountability/Q41B 


Does your state agency provide institutions with feedback on their performance on 
the Perkins accountability measures to permit them to assess their performance 
with others in the state, and if so, what type of information is shared? 


ark all that apply 
Il Statewide performance outcomes averaged across all institutions are shared 
[Performance outcomes are shared, controlling for institutional characteristics 
T Performance outcomes are shared for all individual institutions 
[ Performance outcomes that are shared are posted on the internet for viewing 


[No data on institutional performance are shared 


© Accountability/Q41C 


What changes or additions to the Perkins legislation regarding state or local 
accountability would you recommend that Congress consider for the next 
reauthorization? 
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© Finance/FinanceIntro 


© Finance/Q42 


How much of your Perkins funds were allocated to the secondary and postsecondary 
sectors in your state in each of the following fiscal years? If Tech Prep is merged with 
the Basic Grant, report all funds in Title I Basic Grant. 


Enter dollar amount (whole numbers only, do not include $, commas, or other 
non-numeric characters) 


© Finance/Q43 


If your state changed the percentage split in its allocation of Title I funds to 
secondary and postsecondary education since the adoption of Perkins IV, what 
factors brought about this action? 


Mark all that apply 

T Change in state legislative policies or priorities 

[ Adoption of state economic development initiative 

[Increased focus on academics over CTE instruction at secondary level 
[Increased focus in providing CTE instruction in one sector over the other 
[™ Change in state financing of CTE using non- federal funds 

T Other 

[Not applicable, state has not changed its funding split 


Don't Know 
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© Finance/Q44 
Is your state planning to make any changes to the amounts of funding allocated to 
secondary and postsecondary sectors for the remainder of Perkins IV? 


© Yes, increase secondary sector share of funding 

© Yes, increase postsecondary sector share of funding 
© No 

© Don’t Know 


© Finance/Q45 
How many local applications for 2008-09 program year Perkins IV, Title I funding did 
your state agency receive, and how many of these were approved or disapproved? 


Enter number (whole numbers only, do not include commas or other non-numeric 
characters), mark "Don’t Know” if you do not have this information 


Number disapproved 


© Finance/Q46 
What were the principal reasons for disapproval? 


Mark all that apply 

[No description or inadequate description of local plans for allocating funds 

[No description or inadequate description of local plans for collecting accountability data 
[Failed to provide accountability data in the past 

T™ Poor past performance 

[Failed to agree on negotiated performance target in the past 

[Program determined to lack sufficient quality 

T Other 
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© Finance/Q47_postsec 

What amount of your state’s 2008-2009 Perkins IV, Title I postsecondary funds 
went to each of the following types of providers? [Note: if your state consolidated 
its Title II Tech Prep funds into its Title I Basic Grant, report all funds in the Title I 


category] 


Enter dollar amount (whole numbers only, do not include $, commas, or 
other non-numeric characters), mark "Don't Know” if you do not have this 


information 


colleges 
Other (specify) 


© Finance/Q48_postsec 
Did your state allocate postsecondary grants in 2008-09 under a waiver of the 
required Perkins formula (i.e., Section 132 (b) of the Perkins I'v Act)? 


© Yes 
No 


© Finance/Q48_upload 


Please attach the waiver formula (for allocation of postsecondary grants in 
2008-09) 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, pdf. If you 
have some other file type, please cal! our help desk at 866-434-4718. 


Please attach the waiver formula (for allocation of postsecondary grants in 2008-09), 
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© Finance/Q49 
For the 2008-09 program year, did your state waive the minimum allocation rule 


($50,000 per substate grantee) for any grantees? /Reference: Section 132(2) (4) and 
132(c) (17 


© Yes 
CNo 


© Finance/Q50_postsec 
How many waivers were granted for each of the following reasons? 


Enter number (whole numbers only, do not include commas or other non-numeric 
characters), mark “Don't Know" if you do not have this information 


Institution is in a rural, sparsely populated area 


© Finance/Q51_postsec 
Did your state use the special rule for a minimal allocation of Perkins IV, Title I 


funds at the postsecondary level in the 2008-09 program year? /Reference: Section 133 


(a) (2) and (2)? 
Note: Perkins IV permits 2 state that allocates fess than 15% of its basic grant funds to the 
postsecondary sector to allocate resources using 2 competitive process or alternative method 


determined by the state agency. 


© Yes 
No, were eligible but did not use the rule for minimal allocation 


No, were not eligible to use the minimal allocation 


© Finance/Q52 
For the 2008-09 program year, did your state use the option to create a reserve 


fund to allocate resources to LEAs through a means other than the statutory 


formula? 


C Yes 
No 
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© Finance/Q53 
How much money was placed into the state reserve fund for distribution to 


postsecondary institutions for the 2008-09 program year? 


Enter dollar ammount (whole numbers only, do not include $, commas, or other 
non-numeric characters ), mark “Don't Know" if you do not have this inforrnation 


Know 
Reserve fund amount 


© Finance/Q54 
What amount of your reserve funds were awarded to postsecondary institutions 


located in the following areas? 


Enter dollar ammount (whole numbers only, do not include $, commas, or other 
non-numeric characters), mark "Don't Know" if you do not have this information 


Rural areas 


Other (specify) | 


© Finance/Q55 
How many tribally-controlled or Bureau of Indian Affairs (BIA)-funded LEAs received 


Perkins grants in 2008-09 as noted? 


Enter number (whole numbers only, do not include commas or other non-numeric 
characters), mark "Don't Know” if you do not have this information 


As an individual agency 


wo we ew ewe eee eee ew wwe eee wo ne en ee ee nn ne ne ee ee ne ee ne pee ee eee ee ee ~~ ~~ + ~~~ ---- oe ee ee ee eee eee +- woeeeeee 


As a member of a consortia Cr 
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© Finance/Q56 

Of the funds used at the postsecondary level, on average, what percentage of your 
state’s 2008-09 State Leadership Funds were allocated for the following 
required uses referenced in Section 124 of the Perkins Act? 

[NOTE: Percentages need not total to 100%] 


Enter percentage (whole numbers only, do not include % or other non-numeric 
characters) or mark “Don’t Know” if you do not have this information 


D fa} n' 
Enter Percentage ais 


Strengthening the integration of academic and CTE 
instruction 

Preparing individuals for nontraditional employment 
Supporting partnerships among secondary, postsecondary, 
adult education, and other local institutions 

Serving individuals in state institutions, such as state 
correctional and institutions serving disabled students 
Supporting programs that prepare special education 


students for entry into high skill, high wage, high demand 
occupations 


Technical assistance for eligible recipients 


© Finance/Q57 

How much of the 10 percent State Leadership funding was made available in 
2008-09 for postsecondary services that prepare individuals for nontraditional fields 
(i.e., of the not less than $60,000 and not more than $150,000 earmarked for the 


purpose)? 


Enter dollar amount (whole numbers only, do not include $, commas, or other 
non-numeric characters), mark "Don’t Know” if you do not have this information 
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© Finance/Q58 
What percentage of your State Leadership funds for postsecondary education in 
2008-09 was used to support POS development or implementation? 


Enter percentage (whole numbers only, do not include % or other non-numeric 
characters) or mark “Don’t Know” if you do not have this information 


POS implementation 


© Finance/Q594, 


Indicate whether your state has a gender equity coordinator at the postsecondary 
level and the job status of their position. 


© None 
© Full-time 
© Part-time 


© Finance/Q59B 


Indicate whether your state has a special populations coordinator at the 
postsecondary level and the job status of their position. 


© None 
© Full-time 
© Part-time 
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© Finance/Q60 


From what funding source(s) were the gender equity and special populations 
coordinator positions funded in 2008-09? 


Mark all that apply 

[Perkins State Leadership funds 
[Perkins State Administration funds 
T™ Non-Perkins State funds 

T Other 


© Finance/Q60B 
What changes or additions to the Perkins legislation regarding finance would you 
recommend that Congress consider for the next reauthorization? 


© Finance/Q61_INTRO 


© Finance/Q61 


Did your state provide categorical funding for CTE services offered in postsecondary 
institutions in the 2008-09 program year? 


© Yes 
No 
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© Finance/Q61_upload 
Please attach state CTE funding formula or web link 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, .pdf. If you 
have some other file type, please call our help desk at 866-434-4718. 


Please attach state CTE funding formula 


or provide a web link: | 


© Finance/Q62 
What percentage of the categorical statewide support for postsecondary CTE 
offered in your state in 2008-09 would you estimate came from the following 


sources: (Must total 100 %) 


Enter percentage (whole numbers only, do not include % or ather nan-numeric 
characters) or mark “Don’t Know” if you do not have this information 


Federal sources (excluding Perkins) 


State sources 


Other (specify) | | 


Total 
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© Finance/Q63 
Does your state use performance-based allocations (¢.g., rewards and/or 
sanctions) for state postsecondary CTE funds? 


C Yes 
No 


© Finance/Q63_upload 


Please attach description of state performance-based funding allocation formula or 
web link 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, .pdf. If you 
have some other file type, please call our help desk at 866-434-4718. 


Please attach description of state performance-based funding allocation formula 
or provide a web link: | 


© Tech Prep/TechPrepIntro 
Th tio 


© Tech Prep/Q64 
Did your state choose to merge its Tech Prep funds into its Title I basic grant during 
or prior to the 2009-010 program year? 


C Yes 
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© Tech Prep/Q65 


What are the reasons that your state opted to merge its Tech Prep funding with its 
basic grant funds? 


Mark all that apply 


T Desire to incorporate Tech Prep into all CTE programs 

T Due to the similarity between Tech Prep and POS 

T Avoid data burden associated with collecting new Tech Prep measures 
TC Difficulty identifying Tech Prep students from other CTE students 
Difficulty collecting data on Tech Prep students 


T other 


© Tech Prep/Q66 


Is your state planning to consolidate its Tech Prep funds into its Title I basic grant 
prior to the end of the Perkins IV Act, and if so, when? 


© State has no plans to consolidate funding 
© 2010-11 program year 

© 2011-12 program year 

© Don't know 


© Tech Prep/Q67 


How did your state allocate grants to local Tech Prep Consortia in 2008-09 from 
funds not merged? 


© Through a competitive application and award process 
© By formula 

© Both competition and formula 

© Other (specify) 
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Please attach the guidelines or criteria used to allocate grants to local Tech Prep 
Consortia in 2008-09 from funds not merged. 


Provide your file in one of these formats: .doc, .xls, .csv, .dat, .txt, .pdf. If you 
have some other file type, please call our help desk at 866-434-4718. 


Please attach the guidelines or criteria used. 


© Tech Prep/Q68 
How many local applications for 2008-09 program year Perkins IV, Title II Tech Prep 
unding did your state agency receive, and how many of these were approved or 


isapproved? 


inter number (whole numbers only, do not include commas or other non-numeric 
haracters), mark "Don't Know” if you do not have this information 


Number disapproved 
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© Tech Prep/Q69 
What were the principal reasons for disapproval? 


Mark all that apply 
[No plan submitted or inadequate plan for local allocation of funds 


[No plan submitted or inadequate plan for collection of accountability data 
T Poor past performance 

IT” Program determined to lack sufficient quality 

T™ Other 


© Tech Prep/Q70 

Which, if any, of the following obstacles does your state face in tracking Tech Prep 
students as they transition between the secondary and postsecondary education 
sectors? 


Mark all that apply 
T™ Secondary and postsecondary sectors do not share a common student identifier 


IT” Secondary and postsecondary sectors do not have a means of sharing data 


T™ Secondary Tech Prep students do not self-identify (or know they are a Tech Prep student) 
when matriculating at a postsecondary institution 

[Postsecondary institutions do not attempt to identify incoming secondary Tech Prep 
students 

T Other 
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© Tech Prep/Q71 

The 2006 Perkins introduced a new set of accountability measures for Tech Prep 
students. Indicate your level of agreement or disagreement with each of the 
following statements: 


-qesumeawes 
- + -4ewewcnces 


The new measures will : 
support program 
improvement efforts 
undertaken at the 


State level 


The existence of the 
new measures will 

likely cause my state 
to merge its Title I and: 
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nonregulatory 
guidance to support; 
my state in developing : 
Tech Prep measures’ | 
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® END 

Please select the “validate” button located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and 
that there are no errors present in your responses. If there are errors present, 
simply select the question from the validation summary list to take you directly to 
the question to correct your data. Once you have corrected the data, go to the 
navigation panel on the left side of the screen and select the form titled END. 


You have completed the Survey of State Directors of Secondary Career-Technical 
Education. 


Thank you very much for taking the time to participate in this important study. 


If you have any questions or comments about this study, please contact Steven 
Klein (MPR) at 503-963-3757. 
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© Welcome/Contact_info 


Welcome to the Web-based National Assessment of Career and Technical 
Education survey (NACTE) sponsored by the U.S. Department of Education. 


Please take a moment and complete the contact information below before 
proceeding with the survey. 


First Name: (——$ | 
Last Name: .......... 
Title: [il 
Phone (XXX-XXX-XXXX):[.—“‘“CSCSCOOC*™” 
Email: | 


Address: 


Address 2 (optional): 


© Welcome/Confidentiality 


Uses of the Data: 

This study is being conducted for the U.S. Department of Education, which has contracted 
with MPR Associates, Inc., and RTI International (RTI). The purpose of this study is to collect 
information relating to the implementation of the Carl D. Perkins Career and Technical 
Education Act of 2006 during the 2008-09 academic year. Because a limited number of 
institutions have been selected, we would very much like for you to participate on behalf of 
your educational institution. 


Confidentiality 

As a matter of policy, The U.S. Department of Education is concerned with protecting the 
privacy of participants in voluntary surveys. 

We want to let you know that: 


Your responses will be merged with those of other respondents and will not be 
identified as the agency you represent, except as required by law. 

You may skip questions you do not wish to answer; however, we encourage you 
to answer as many questions as possible, because incomplete data will reduce the 
value of the information provided to Congress. 

Your answers may be used only for statistical purposes. Participation is voluntary 
and will not affect any aid or other benefits that your state agency, LEA, or 
institution may receive. 


If you have questions about the study, you may contact: 


the NACTE Help Desk at (866) 434-4718 

Steven Klein, the study's director, at (503) 963-3757 or sklein@mprinc.com, or 
Michael Fong, the U.S. Department of Education's Project Officer, at (202) 401-7462, 
Michael Fon: -gov. 


If you have questions about your rights as a study participant, you may contact RTI's Office of Research 
Protection toll free at (866) 214-2043. 
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© Welcome/Definitions 

Definitions: 

Some questions in the survey include a hyperlink (a word or group of words in blue font that is underlined) 
that, when clicked on, brings up a definition. 


For example, 
|Career and Technical Education (CTE): The term ‘career and technical education (CTE)’ means organized 
educational activities that— 
(A) offer a sequence of courses that— 

{i) provides individuals with coherent and rigorous content aligned with challenging academic standards 
and relevant technical knowledge and skills needed to prepare for further education and careers in current or 
emerging professions; 

(ii) provides technical skill proficiency, an industry-recognized credential, a certificate, or an associate 
degree: and 

(iii) may include prerequisite courses (other than a remedial course) that meet the requirements of this 
subparagraph; and 

(B) include competency-based applied learning that contributes to the academic knowledge, higher-order 
reasoning and problem-solving skills, work attitudes, general employability skills, technical skills, and 
occupation-specific skills, and knowledge of all aspects of an industry.including entrepreneurship, of an 
individual. 


Programs of Study (POS): Programmatic coursework that meets the following: incorporates both 
secondary and postsecondary education elements; includes coherent and rigorous content aligned with 
challenging academic standards and relevant career and technical content in a coordinated, non-duplicative 
progression of courses that align secondary education with postsecondary eduction to adequately prepare 


students to succeed in postsecondary education; may include the opportunity for secondary education 

students to participate in dual or concurrent enrollment programs or other ways to acquire postsecondary 

education credits: and leads to an industry-recognized credential or certificate at the postsecondary level, or 
jan associate or baccalaureate degree. 


© Welcome/Navigation_Instructions 


Instructions and Navigation: 
NACTE Help Desk: 866-434-4718 


To navigate through the survey you will notice two buttons in the lower left hand corner of each screen titled 
“Previous” and “Next”. You can select either button at anytime to move through the survey. 


Alternately, you can also use the navigation panel located on the left-hand side of the screen. You may jump 
to any question in navigation panel simply by clicking on the question number. 


The questions in the navigation panel will turn green once they have been viewed/answered and 
have passed preliminary validation. 

If a question does not pass preliminary validation either because of incomplete information or 
errors with the response, the question in the navigation panel will turn yellow. 

Questions in the navigation menu that are grey, and have a strike through are not applicable to 
you (e.g., Q62). 


The “Save & Exit” button located in the lower right-hand corner of the screen allows you to exit the survey at 
any time. The next time you login to the survey you will return to the same question. 


Also, located in the lower right-hand corner of the screen is the “Help” button. It contains information 
regarding recommended browsers, and contains definitions for the survey response items (i.e., drop-down 
boxes, checkboxes, and hyperlinks). 


At the end of the survey, please select the “validate” button located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and that there are no errors 
present in your responses. 
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© Programs of Study/Q1 
Did your institution receive a federal Perkins Title I and/or Title II grant in the 


2008-09 program year? 


Title I: Basic Grant 
Title II: Tech Prep 


© Programs of Study/Q2 
Did your institution receive a federal Perkins Title I and/or Title II grant through 


participation in a consortium in the 2008-09 program year? 


Title I: Basic Grant 


Title II: Tech Prep 
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© Programs of Study/Q3 
How many Programs of Study (POS) offered in your institution in the 2008-09 
program year were developed at the state or local level? 


(Mark "Don't Know” if you do not have this information) 


Locally developed POS, using state template or guidance 
and... 


State approval required and obtained : 
State approval required and not yet obtained 
State approval not required 


Locally developed POS without state a and... 


© Programs of Study/Q4 
Are any of the POS offered in your institution described on a Web site? 


C Yes 
™No 


© Programs of Study/Q4_url 


Please provide the web address (url) of a website describing one of the POS offered 
in your institution. 


What is the URL? | 
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© Programs of Study/Q5 


In the following table, please list the five (5) POS with the highest enrollments in you 


institution in the 2008-09 program year and the number of students in each. If you 
have less than five POS then please list them al Pas 03 ac chi San) POS thal 


Type in the name of the POS in the first box, code it into the corresponding Career 
Cluster area (or closest corresponding area) by clicking on the second box, and ente 
the number of students in the third box. Mark ‘Don’t Know’ if you do not have acces: 
to this information. 


Number of students 
enrolled in coursework 
in the POS in the 
2008-2009 program 
year 


Name of state-approved POS 
NOTE: Please crosswalk your POS to the appropriate cluster using 
the pull-down menu provided 


Agricultural, Food, and Natural Resources 
Architecture and Construction 

Arts, Audio/Video Technology and Communications 
Business Management and Administration 
Education and Training 

Finance 

Government and Public Administration 


Health Science 


Hospitality and Tourism 

Human Services 

Information Technology 

Manufacturing 

Marketing 

Public Safety 

“Science, Technology, Engineering and Mathematics 
Transportation, Distribution and Logistics 
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© Programs of Study/Q6A 
How many of the five POS with the highest enrollments (listed in Q5) incorporated 
each of the following characteristics of a POS? 


Standards 


(Select "Don’t Know” if you do not have this information) 


Aligned with technical | 
standards that were... | 
a. state-developed : 


c. national based on 
16 career clusters ' 


e. other (specify) 
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© Programs of Study/Q6B 
How many of the five POS with the highest enrollments (listed in Q5) incorporated 
each of the following characteristics of a POS? 


Curriculum 


(Select "Don't Know” if you do not have this information) 


d. locally developed ! | 


e. other (specify) | 


Uses a curriculum that! 
spans secondary and: 
postsecondary levels: 


Is part of an 

articulation agreement : 

with ONLY one 
secondary district ; 


(in-state or out-of- 
state) 

Is covered by a : 
state-wide articulation : 
agreement 


credit to secondary: 
students through (dual 


or concurrent credit) | 
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© Programs of Study/Q6C 
How many of the five POS with the highest enrollments (listed in Q5) incorporated 


each of the following characteristics of a POS? 
Assessments 
(Select "Don't Know” if you do not have this information) 


d. state licensing or 
credentialing exams 


(ii) aligned with 
national 16 career 
cluster standards ‘ i 


1 
opeccesesces 
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© Programs of Study/Q6D_postsec 
How many of the five POS with the highest enrollments (listed in Q5) incorporated 
each of the following characteristics of a POS? 


Credentials 


(Select "Don't Know” if you do not have this information) 


' 
' 
' 
' 
' 
' 
' 
' 
' 
. 
' 
. 
' 
. 
' 
' 
' 
r 
1 
1 
1 
1 


Postsecondary : 
level, ' 
including... 

@ 
certificate 
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© Programs of Study/Q6E 


How many of the five POS with the highest enrollments (listed in QS) incorporated 
each of the following characteristics of a POS? 


Other Features 


(Select “Dan't Know" if you do not have this information) 


Responds to : 
high skill, ' 
high wage, ! 
high demand : 
occupational : 
areas : 
identified by : 
the state or | 
LEA : 


face en nw en en ee eee ee nee ee 
ee 


Career 
guidance 
must be, or ! ' ‘ ‘ ' ' 
is, available : ‘ ; ; 


Was 
previously 

part of Tech! 
Prep ' 
program ' 
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© Programs of Study/Q7A 


On which of the following topics did your state offer technical assistance to help 
institutions in your state in developing POS for state-approval? 


(Mark all that apply) 

[POs templates or guidelines 

Tl Career clusters 

[CTE content standards 

[CTE curriculum development guidelines 

T Academic and CTE curriculum integration 

T™ Secondary and postsecondary curriculum alignment 
[Technical skill assessments 

[ Career guidance and counseling 

[Data systems for monitoring student progress 
T Aligning standards and assessments 


TC Other 
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© Programs of Study/Q7B 
In which of the following areas (identified in Question 7A) did your institutional staff 
participate during the 2007-08 or 2008-09 program years? 


(Mark all that apply) 

[Pos templates or guidelines 

Tl Career clusters 

[CTE content standards 

[CTE curriculum development guidelines 

T Academic and CTE curriculum integration 

T™ Secondary and postsecondary curriculum alignment 
[Technical skill assessments 

[ Career guidance and counseling 

[Data systems for monitoring student progress 
[ Aligning standards and assessments 

T Other (if specified) 
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© Programs of Study/Q8 
What types of professional development does your state offer AND in what types of 
activities did your institutional staff participate in to support the development and 


implementation of POS? 


(Mark all that apply for each group. If your state offered other types of professional 
development, specify the type(s) of professional development in textbox(es) in the 


“Other” column below.) 


State/ 
regional 
conferences 
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© Programs of Study/Q9A_postsec 


To what extent did each of the following stakeholder groups participate in your 
institution's efforts to develop POS? 


Postsecondary academic instructors 


Postsecondary CTE instructors 


National industry/union groups or 


ie. 
ie 
2) 
ie 


{ 
--- 
1 


1°) 
e) 


i.) 
ie 


Institution of Higher Education (IHE) Survey 14 of 33 


we Reference copy 
 eyY NACTE Please complete 
estilo pie tee art oer 


© Programs of Study/Q9B_postsec 


To what extent did each of the following stakeholder groups participate in your 
institution's efforts to implement POS? 


Postsecondary academic instructors 


O my 
oO mo 


Postsecondary CTE instructors 


' 
1 
‘ 

i sete 
¥ 
' 


‘e 


ie 
@) 
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© Programs of Study/Q10A 


To what extent has each of the following barriers limited your institution's efforts to 
POS? 


a 


Shortage of institutional CTE staff 


oO my 
oO 
oO Bo 


Shortage of state CTE staff 


-- faeea= -- - 


--P====}-- 


1 

1 
a 

' 

1: 

1‘ 

. 
~ 

1 

1 
ate 

1 

' 

' 

' 

. 
- 

1 


Lack of cooperation from secondary 
teachers or administrators 


Lack of cooperation by state level staff 


-<+= ===}--- 


Resistance to state influence at local 


{ee === ----- 


-p====}$---- 


---+ $-===4---- 


@) 
oO 
'@) 
1”) 


Lack of state leadership 
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© Programs of Study/Q10B 


To what extent has each of the following barriers limited your institution's efforts to 
POS? 


Shortage of institutional CTE staff 


Shortage of state CTE staff 


Lack of curricular materials 


Lack of Se decniant Peteumante 


% 
% 


Lack of technical expertise 


(ey e eeeruen by CTE “Eapiee 


ie 
4 


Lack of cooperation by administrators 


Lack of cooperation from secondary 
teachers or administrators | 


ie, 
ary 
e 


Oo 
% 
1°) 


Lack of cooperation by sels ne staff 


Resistance to state influence at local 


deoneot-- 


Shortage of time 


Te’ 


Conflicts beteecn ralosne jpiuduels 


. 
a a cae ea eel ace PIRI M ere 
1 1 


Cannot find qualified instructors 


WORE O 
tO my 


pack, of See snea Ron the anes 


% 
oO 


Lack of state leadership 
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© Accountability/ACC_INTRO 


The questions in the remainder of this survey address all of the CTE 
——————— pective of whether or not they 


© Accountability/Q11_postsec 

How difficult was it for your institution to reach agreement with your state CTE 
agency on performance benchmarks and targets for each of your Perkins 
accountability measures? 


Attainment of an : 
industry- 
recognized 
credential 
certificate, or 
degree (2P1) 


ee 
F eee eee ee 


Placement into 2 an! 
apprenticeship 
program, the 
military, or 
employment (4P1) | 


_ 


peneeeee-4 
fonnnnneo ed 


Non- Sradional 
completion (5P2) 
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© Accountability/Q11B 


When negotiating with your state education agency to establish annual performance 
benchmarks and targets for your institution, what were the reasons for 
encountering difficulty? 


(Mark all that apply) 

T Insufficient baseline data upon which to negotiate targets 

IT Lack of clear guidance from the state on what constitutes continuous improvement 

[ Anticipated difficulty in gathering and submitting required data 

T anticipated difficulty in being able to meet proposed performance targets 

[™ Concern over the repercussions of your institution failing to meet performance targets 


T other 


© Accountability/Q12 


How do you identify students meeting the CTE concentrator threshold for Perkins 
accountability purposes? 


(Mark all that apply) 

T Students self-report concentrator status 

[Faculty identify students based on course taking 

[ Administrators identify students using local management information system 

[ State identifies students using statewide database; no action at institutional level 
T Other 

T™ Don't know 
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© Accountability/Q13 


How do you ensure that the accountability data you collect for the Perkins 
accountability measures are accurate? 


(Mark all that apply) 

[Follow state guidance on measure development 

T™ Provide technical assistance to teachers 

T Require teachers or administrators to review or spot-check data submissions 
T Other 

[No steps taken to ensure data accuracy 


T™ Don't know 
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© Accountability/Q14_postsec 
How confident are you that the data you are collecting for each of the following 


Perkins core indicators of performance provides an accurate measure of your 
institution's actual performance? 


Core Indicator 1 


Technical skill i 
attainment (1P1) : 


Core Indicator 2 


--Continued on next page -- 
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Cereer and Tedinical Edueestion 


--Continued from previous page— 


© Accountability/Q14_postsec 
How confident are you that the data you are collecting for each of the following 


Perkins core indicators of performance provides an accurate measure of your 
institution's actual performance? 


Core Indicator 4 


Placement in 
apprenticeship 


Non-traditional 
completion (5P2) 
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© Accountability/Q15 

How confident are you that the data you are collecting for each of the Perkins 
special population categories provides an accurate measure of the actual 
performance of these populations? 


Individuals from 
economically 
disadvantaged 
families, including : 
foster children 


Single parents, 
including single 
pregnant women 


Individuals with 
limited English 
proficiency 
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© Accountability/Q16A 
In what ways and to what extent did your institution use its prior year Perkins 
accountability data in the 2008-09 program year? 


Used not at} Used not Used Used quite 
all much | somewhat a bit 


To make program 
funding decisions 


To identify special 
population students not : 
being adequately served : 


Overall, how useful are 
the Perkins data you 
collect for your local 
program improvement 
efforts? 
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© Accountability/Q16B 


In the previous question you indicated that your institution did not use its prior year 
Perkins accountability data in the following ways in the 2008-09 program year. Are 
there plans to use the data for any of these purposes in the next two years? 


To identify programs in need of improvement 


To make program funding decisions 


To identify special population students not being 
adequately served 


© Accountability/Q17A 


What type of feedback does your state agency provide you on your institution's 
performance on the Perkins accountability measures? 


(Mark all that apply) 


[No data on my institution performance is shared 


[ Statewide average performance outcomes for all institutions are shared 


T™ Performance outcomes are shared for institutions with characteristics similar to my own 


[Performance outcomes are shared for all individual institutions 
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© Accountability/Q17B 


If your institution were to fail to meet its Perkins performance targets, does your 
state have a policy for the type of corrective action you must take to address the 
accountability deficiencies? 


(Mark all that apply) 
[Not applicable, no state policy exists 


T™ Develop a program improvement plan 


T Provide professional development to staff of underperforming programs 


T Consult with state or local business or industry experts in the field 
[Eliminate underperforming programs 

T Restrict student access to programs 

[ Restructure or redesign existing CTE programs 


Tl Other (specify) 


T™ Don’t know 
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© Accountability/Q18 

What proportion of the students participating in state-approved CTE programs withi 
your institution were assessed using one of the following methods to measure their 
technical skill attainment in the 2008-09 program year? 


National licensing or 
credentialing exams 


Commercially developed 
exams (e.g., NOCTI) 


Grade point average (GPA): 


Course or program completion 
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© Accountability/Q20 

The 2006 Perkins Act introduced a new set of accountability requirements for Tech 
Prep students. Indicate your level of agreement or disagreement with each of the 
following statements: 


Somewhat Somewhat} Strongly 
Disagree Disagree Agree piste 


The new measures will ; 
support program 
Spit efforts at! 


the State level | 


support program 
improvement efforts at 


Data collected for the | 
measures are of high 


My institution is able 
to collect data on all 
measures 
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© Finance/Q21 
How much of the total budget for CTE offered in your institution in the 2008-09 
program year came from each of the following sources? 


(If none, enter "0”, Mark “Don’t Know” if you do not have this information) 


State funds: 


Formula (must be spent on CTE) ; i 


Tech Prep Title II 


Other federal funds (specify) | ' 
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© Finance/Q22 


For which of the following permissible uses referenced in Section 135(c) of the 
2006 Perkins Act did you use your 2008-09 Title I basic grant funds? 


(Mark all that apply) 


[Involving parents, business and labor in designing, implementing, and evaluating CTE 
programs covered by the Act 
T™ Providing career guidance and academic counseling 


T™ Promoting work-related experiences for students 

T™ Promoting industry experiences for teachers 

[™ Providing programs for special populations 

[ Assisting CTE student organizations 

T Offering mentoring and related support services 

[ Leasing, purchasing, upgrading, or adapting equipment 
T Supporting teacher preparation programs 
[Entrepreneurship education and training 

T™ Developing new CTE courses 

T Creating small, personalized career- themed learning communities 
T Supporting family and consumer sciences programs 
Offering programs for adults and school dropouts 
[Offering continuing education or job referral services 

T Supporting nontraditional training and activities 

T™ Providing training programs in automotive technologies 
[Improving accountability data collection and reporting 
[Implementing CTE programs of study 


Tl Other 
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© Finance/Q23 


What impact has each of the following changes introduced in the 2006 Perkins Act 
had on the administration and implementation of CTE programs in your institution? 


ee eee eee 
eee eee 

+ fe mw mmc ecwcen 
errs 


Requirement to 
dedicate funds to 
support students, 
including special 
populations, in 
preparing for high: 
wage/skill/demand : 
jobs : 


New allowable use : 
of funds to form 
consortia to 
maximize 
investment and 
encourage 
innovation 


1 1 
. 
1 1 
1 1 
1 1 
1 1 
1 1 
1 1 
. 
a ee 
' ' 
' 
' 


Requirement to 
negotiate 
performance 
benchmarks with 
the State agency |: 
to qualify for funds : 


wn ne fe ewww ew meee ewe wnnd--------- 
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$50,000 per substate grantee) in 2008-09, why was it granted? [Reference: Section 
32(a) (4) and 132(c) (1)] 


© No waiver granted 

© Institution is in a rural, sparsely populated area 
© Other (specify) 

© Don't know 


© END 

Please located in the lower right-hand corner of the 
screen. The validate button will ensure that all questions have been answered and 
that there are no errors present in your responses. If there are errors present, 
simply select the question from the validation summary list to take you directly to 
the question to correct your data. Once you have corrected the data, go to the 
navigation panel on the left side of the screen and select the form titled END. 


You have completed the Survey of Institutions of Higher Education Offering Career- 
Technical Education. 


Thank you very much for taking the time to participate in this important study. 


If you have any questions or comments about this study, please contact Steven 
Klein (MPR) at 503-963-3757 
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Appendix B. Programs of Study 
Expert Panel Review 


Introduction 


The Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV) requires that local 
suberantees offer one or more programs of study (POS) to link career and technical educa- 
tion (CTE) offered at the secondary and postsecondary education levels. POS are intended 
to provide students with access to sequenced, nonduplicative coursework, aligned with chal- 
lenging academic standards and rigorous technical content, that leads to the conferral of an 
industry-recognized credential or certificate at the postsecondary level or an associate’s or 
bachelor’s degree. Where practical, high school students also can acquire postsecondary edu- 


cation credit through participation in dual credit or concurrent enrollment programs. 


Perkins IV provides no direction on how local subgrantees are to meet this requirement, 
leaving it to state education agencies to establish criteria for compliance. To assess the guid- 
ance provided by states for the development of POS, the research team convened an expert 
panel in April 2010 to review materials collected from 50 state education agencies. The 10 
expert panelists included secondary and postsecondary state administrators and representa- 


tives of national stakeholder groups and policy and research firms. The panelists are listed in 
Appendix A. 


To structure the POS review process, the research team created a scoring rubric based on 
the POS Design Framework (Framework), released by the U.S. Department of Education (De- 
partment) in January 2010.' Because the design framework was released several years after 
the passage of Perkins IV, states did not have access to the information contained in the 
Framework when developing state guidance. The Framework consists of 10 components 
that, taken together, promote the development and implementation of effective POS. These 


components include the following. 


e = Legislation and Policies—Federal, state, and local legislation or administrative poli- 


cies promoting POS development and implementation. 


! http://cte.ed.gov/nationalinitiatives /rposdesionframework.cfm (accessed 9/24/2010). 
tp g P gn. 
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e = Partnershis—Ongoing relationships among education, business, and other 
community stakeholders that are central to POS design, implementation, and 


maintenance. 


e = Professional Development—Sustained, intensive, and focused opportunities for ad- 
ministrators, teachers, and faculty involved in the design, implementation, and 


maintenance of POS. 


© =§=Adccountability and Evaluation—Systems and strategies to gather quantitative and 
qualitative data on both POS components and student outcomes to aid ongoing 


efforts to develop and implement POS. 


e = College and Career Readiness Standards—Content standards defining what students 
ate expected to know and be able to do to enter and advance in college and/or 


careers. 


© Course Sequences—Nonduplicative sequences of secondary and postsecondary 
courses within a POS ensuring that students transition to postsecondary educa- 


tion without duplicating classes or requiring remedial coursework. 


© = Credit Transfer Agreements—Credit transfer agreements that provide opportunities 
for secondary students to accrue transcripted postsecondary credit, supported 


by formal agreements between secondary and postsecondary education systems. 


© Guidance Counseling and Academic Advisement—Guidance counseling and advise- 


ment helping students to make informed decisions about which POS to pursue. 


© §=6Teaching and Learning Strategies—Innovative and creative instructional approaches 
enabling teachers to integrate academic and technical instruction and students to 


apply academic and technical learning in their POS coursework. 


© = Technical Skill Assessments—National, state, and/or local assessments providing 
ongoing information on student attainment of the necessary knowledge and 
skills for entry and advancement in postsecondary education and careers in their 
chosen POS. 


Researchers added an 11th component, asking panelists to provide an overall rating of the 
Clarity and Coherence of each state’s POS guidance. The rubric included a definition of key 
terms, examples of the type of guidance that might exist under each of the 10 Framework 
components, and space for panelists to describe and rate the usefulness of state materials in 


each category. The rubric is included at end of this appendix. 
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Findings 


Panelists found that all states had addressed, either directly or indirectly, at least some com- 
ponents of the Framework. Because the Framework was not released until early 2010—more 
than three years following the passage of Perkins [V—panelists’ ratings reflect the extent to 
which states successfully anticipated and addressed areas subsequently identified by the De- 


partment as important to the development and implementation of effective POS. 


During webinar discussions, panelists reported that some states appeared to have invested a 
considerable amount of time and staff resources in creating POS guidance and materials for 
local provider use. These states had created informative, user-friendly websites that included 
information on most, if not all, of the 10 components. Other states appeared to have just 
begun to develop POS guidance materials and make them available through the state website 
ot documents distributed to locals. Materials in those states, while potentially useful to local 
providers, were rated by the expert panelists as providing relatively less guidance or direction 
on the design and implementation of POS.? Panelists suggested a variety of explanations for 
the lack of information, including the possibility that resources were lacking for the devel- 


opment of materials and websites to offer guidance. 


Average State Ratings by Component 


The expert panelists categorized state materials into one or more of the 10 Framework com- 
ponents, using a rating scale of 1-4 (with 4 denoting the clearest, most useful guidance). Av- 
erage component ratings ranged from a low of 2.2 to a high of 3.0, with the overall clarity 
and coherence of POS guidance across components and across states rated at 2.8 (Exhibit 
B.1). 


? While research staff sought to ensure that panelists had access to all state guidance and supporting 
materials, it is possible that state materials sent to local providers via hard copy or other means were 
not shared with study researchers. It is also possible that materials posted soon after the Perkins IV 
reauthorization were subsequently removed from websites and thus were not included in this review. 
Caution should therefore be used in drawing conclusions about the overall status or quality of materi- 
als developed by states. 
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Exhibit B.1. 
Avetage panel ratings of usefulness of states’ POS guidance 


Average rating of materials 
4 = Very useful 


3 = Useful 
Relative POS component 2 = Somewhat useful 
rating (in order of highest to lowest average rating) 1 = Not useful 
High Legislation and Policies 3.0 
Course Sequences 2.8 
College and Career Readiness Standards 2.8 
Guidance Counseling and Academic Advisement 27 
Medium Professional Development 2.6 
Accountability and Evaluation 2.5 
Partnerships 2.5 
Low Credit Transfer Agreements 2.3 
Teaching and Learning Strategies 23 
Technical Skill Assessments 2.2, 
Overall clarity and coherence 2.8 


Exhibit reads: Panelist members rated the usefulness of states’ POS guidance differently for the various POS components, with the 
highest average rating assigned to Legislation and Policies and the lowest to Technical Skill Assessments. 


Some POS components were clearly given a low numeric rating by most panelists (e.g., 
technical skill assessments) or a high one (e.¢., legislation and policies) (Exhibit B.2).° For 
other components, however, ratings across panelists were mixed, with similar proportions 
rating them as very useful or useful and not at all or somewhat useful (e.g., partnerships, ac- 
countability and evaluation). This distribution of panelists’ ratings on the POS components 


clearly affected the average ratings. 


5 Panelists were paired in their review of state materials, with each state’s guidance materials rated by 
each panelist. As such, a total of 102 possible ratings were possible (2 for each of 51 states); however, 
since not all reviewers rated each state, total number of ratings assigned to states’ guidance materials 
may total less than 102. 
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Exhibit B.2. 
Number of panelists assigning a rating to each POS component, by degree of usefulness 


@ Very useful @ Useful @ Somewhat useful Not at all useful 


POS component 
Technical Skill Assessments 


Teaching and Learning Strategies 


Credit Transfer Agreements 


Partnerships 

Accountability and Evaluation 

Professional Development 

Guidance Counseling and Academic Advisement 


Career and College Readiness Standards 


Course Sequences 


Legislation and Policies 


0 20 40 60 80 100 


Number 


Exhibit reads: The number of panelists rating states’ POS guidance as useful was highest for Legislation and Policies, with 41 
rating guidance as very useful and 30 as somewhat useful; ratings for other framework components varied, with the lowest ratings of 
usefulness assigned to Technical Skill Assessments. 

NOTE: Total possible ratings is 102 (2 ratings per state). Numbers may not sum to 102 due to missing ratings. 


Overall, the components of states’ POS guidance with the highest average ratings were Leg- 
islation and Policies, Course Sequences, College and Career Readiness Standards, and Guid- 
ance Counseling and Academic Advisement. Components with the lowest average ratings 

wete Credit Transfer Agreements, Teaching and Learning Strategies, and Technical Skill As- 


sessments. 


Challenges in State Guidance on POS 


In reviewing the guidance materials, panelists noted a number of issues that compromised 
states’ efforts to support local agencies in designing and implementing quality POS. The key 


challenges are described below. 


Disorganized websites 
Panel members noted that many local providers use the Internet to search for information 
on POS; therefore, they suggested that states make an effort to include all POS guidance ma- 
terials online, preferably in one frequently updated website, with a clear site map and user- 
friendly navigation tools. Because POS are intended to span secondary and postsecondary 


education, instructors at both levels would benefit from being able to access information on 
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POS from a single location, rather than having to seek information separately on either the 


secondary or postsecondary education agency website. 


Missing information 
In some instances, panelists encountered state education websites that had missing or frag- 
mented information or links that no longer worked, which prevented them from accessing 
information. While panelists found templates or guidance on procedures and documents 
(e.g., memoranda of understanding) helpful, they noted that providing instructions (.e., how 
to use the template) next to the documents also would have made it easier to access the in- 
formation. Not unexpectedly, because Perkins IV does not have any reporting requirements 
associated with POS, panelists did not find much information on POS data systems or ac- 


countability. 


Lack of contextual information on POS nationally 
Panelists expected to see more information about the connection between state and federal 
legislation and between legislation and funding. They noted that local program staff might 
have little understanding of the “big picture” or the reasoning behind state decisions without 


a clear explanation offering a rationale and guidelines for local implementation. 


Lack of direction on the four legislated components of POS 
Panelists expressed concern about the absence of direction from states on how local pro- 
grams might address the four elements of POS outlined in Perkins IV. They noted a lack of 
guidance on the progression of courses spanning secondary and postsecondary education 
levels; information on recognized industry certifications and local and state institutions offer- 
ing degrees and certificates for a given POS; and career options for students, such as the 
fastest growing occupations and salary data. Panelists suggested that all states provide POS 
templates for each program/cluster area, as some states have already done. They also sug- 
gested that information be made available for local staff on how to adopt or adapt these 


templates, including specific guidance on the 10 Framework components. 


Lack of POS-specific language 
Panelists commented that state websites mentioned POS less frequently than they had ex- 
pected. The components of POS often were addressed on the CTE web page, but there was 
little reference to POS specifically. Given the similarities between Tech Prep and POS, pan- 
elists expressed surprise at the lack of information on overlapping components, such as sec- 
ondary and postsecondary partnerships, in states that did not merge their Title I and Title H 


funding.* In cases where states had well-developed Tech Prep systems prior to Perkins IV, 


4 Perkins IV establishes two similar approaches that recipients may use for CTE: Tech Prep [Sec. 203 
(©] and programs of study [Sec. 122 (c)(1)(A)]. Both approaches are required to have secondary and 
postsecondary elements, use rigorous academic standards and technical content, and lead to an indus- 
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the panel noted that there appears to have been little effort to convert the language or ap- 
proach to reflect POS. They also commented that in many states, the relationships among 
Tech Prep, Career Clusters, Career Pathways, and POS were unclear; for example, only one 


state provided a glossary and crosswalk for these approaches. 


Less information on postsecondary POS 
Panelists observed that the state postsecondary education agency charged with administering 
CTE appeared to be less involved than the secondary CTE agency in the development and 
dissemination of POS guidance. This conclusion was based on the lack of information on 
state websites—both secondary and postsecondary—for a postsecondary audience. For ex- 
ample, while information on teaching and learning strategies, tools, and resources for teacher 
professional development were rare among states’ POS guidance for the secondary level, 


guidance on these topics for the postsecondary level was absent altogether. 


One panelist suggested that this dearth of information for postsecondary educators may be 
tied to states’ allocations of Perkins funds, which in many cases go mainly to secondary edu- 
cation.> Another reason may be the separation of secondary and postsecondary offices at the 
state level; in some states, the separate secondary and postsecondary CTE websites often 
have no obvious connection to each other. Perhaps reflecting these “silos,” there was little to 
no information in states’ POS guidance about how to develop secondary/postsecondary 


partnerships. Indeed, the average rating for partnerships was relatively low. 


Similarly, panelists found little information in states’ guidance about how to partner with in- 
dustry or use industry standards in the development of POS, resources regarding technical 
skill assessments, or a definition of what it means to be college and career ready. Other than 
career interest tools, panelists also noted that there was little information on career readiness, 
and they indicated that without such information, it might be difficult to reach out to busi- 
nesses or create POS in emerging industry areas. Finally, panelists noted that while online ca- 
reer planning tools were relatively abundant on state websites, they needed to be linked more 
directly to POS. 


Despite these shortcomings, panelists identified materials from 27 states that they agreed 
should be shared with other states because they appeared to have overcome many of the 
challenges cited above. Descriptions of these materials and website links (where applicable) 


ate listed below. 


try-recognized credential, certificate, or postsecondary degree. While Title II Tech Prep has been 
made optional for states, the major features of Tech Prep are sustained as a requirement through POS 
within Title I of the legislation. 

> According to study findings, states allocated an average of 63 percent of funds to the secondary sec- 
tor in fiscal year (FY) 2010. 


APPENDIX B. PROGRAMS OF STUDY EXPERT PANEL REVIEW B-8 


Useful Aspects of States’ POS Guidance 


The major observations of the expert panelists on the useful aspects of states’ POS guidance 
are discussed in the next sections. They are organized according to the 10 Framework com- 
ponents and are ranked from highest to lowest. Each component is defined below, followed 


by the related panel findings. 


Legislation and policies 
These include legislative or administrative policies that establish funding and resources for 
POS development, as well as procedures and guidance for POS implementation and im- 


provement. Legislation and policies provide for long-term POS stability. 
Useful guidance included the following: 


e Clear, searchable guides on state requirements and policies (e.g., funding, dual 
enrollment, or industry certifications) written for a local audience and including 


implementation guidance; 


e Clearly written guidance documents for local implementers, including Frequent- 
ly Asked Questions (FAQs) about federal and state legislation on POS; and 


e Sample local POS development and implementation plans that cite legislation. 


Panelists noted that Ilinois and Ohio offered useful examples of comprehensive guidance 
covering the design, development, and implementation of POS. Examples of these resources 


were noted at: 


e = Illinois: http://occrl illinois.edu/files /Projects/perkins /POSguide.pdf 


e =Ohio: 
http://education.ohio.gov/GD/Templates/Pages/ODE/ODEPrimary.aspx?P 
age=2&TopicID=1752&TopicRelationID=2 


Course sequences 
A clearly defined POS should contain a course sequence to be used as a program road map 
for a student pursuing a particular career path. The sequence outlines both secondary and 
postsecondary courses and, to the degree possible, eliminates any unnecessary duplication of 


coursework between the two educational levels. 
Useful guidance included the following: 


e POS templates for each program/cluster area spanning secondary and postsec- 
ondary levels, with a clear progression that includes dual credit opportunities at 


local and state institutions, industry certifications, and career options. 
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Indiana offers an example of course sequence templates for POS implementation at 


http://www.doe.in.gov/pathways/. 


College and career readiness standards 
College and career readiness is identified through academic and technical curricular content 
to establish what a student is expected to know and demonstrate in order to be prepared for 
advanced education and training as well as career entry. Content development should be a 


collaborative effort among secondary educators, postsecondary educators, and employers. 
Useful guidance included the following: 


e State definitions of what it means to be “college and career ready”; 
e §=Curticulum frameworks and academic and technical skill standards for each POS; 
e Crosswalks of industry, academic, and CTE POS standards; and 


e POS templates illustrating a coherent sequence of academic and technical 


coursework meeting college and career readiness standards. 


Examples of CTE content standards and college and career readiness tools used by Califor- 


nia, Nebraska, New Jersey, and Oklahoma can be found at: 


e California: http://www.statewidepathways.org/ and 
http://www.cde.ca.gov/ci/ct/sf/documents/ctestandards.pdf 


e Nebraska: http://www.nde.state.ne.us/NCE/ 
e New Jersey: http://www.state.nj.us/education/voc/toolbox.htm 


e = =Oklahoma: http://www.okcareertech.org/testing/skills_stands.htm 


Guidance counseling and academic advisement 
A planned POS course sequence may not reflect the actual path a student chooses. Guidance 
and advisement are critical when a student elects to be creative on his or her chosen career 
path. Guidance tools such as a course sequence planner should be made available for the 
student to visualize the POS path. Educators providing guidance and advisement can offer 
counsel to students who may refine their plan over time as they become more focused on 


their career pursuit. 
Useful guidance included the following: 


e Third-party career exploration tools (e.g., The Real Game, which involves role- 
playing in future-based scenarios to help students learn experientially about the 


essential workplace skills needed for different jobs); 
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e Career planning websites and tools; and 


e Separate information for students and parents and guidance counselors and 


school personnel. 


Examples of resources for guidance and advisement can be found on the California, Colora- 


do, and New York websites at: 


e = California: http://www.californiacareers.info / 
e Colorado: http://www.coloradostateplan.com/Counselors.htm 
e New York: https://www.nycareerzone.org/cz/index.jsp 
Professional development 
To mirror POS design, the professional development of educators also reflects both second- 
ary and postsecondary participation and an appropriate mix of academic and technical in- 
structors. Professional development needs to be sustained, intensive, and focused on 


classroom instruction. Administrators and guidance counselors ate encouraged to take ad- 


vantage of POS professional development opportunities. 
Useful guidance included the following: 


e Presentations and videos for use by local directors in teacher training work- 


shops; 
e Local and regional workshops offered by state staff; 
e Technical assistance by state staff; 


e Conferences in which local directors, teachers, and state staff can share ideas 


and information; and 


e Regular communication between state and local staff (e.g., monthly or quarterly 


calls), with agenda and minutes posted. 


Examples of technical assistance resources and FAQs for professional development used in 


Arkansas, Texas, Florida, and Pennsylvania can be found on their websites at: 


e Arkansas: http://ace.arkansas.gov/CareerandTechEducation/presentations.htm 
e Texas: http://cte.tamucc.edu/la/online_library.shtml 


e = Florida: http://www.fldoe.org/workforce/technicalassistancepapers.asp 
http://www. fldoe.org/workforce/conf-call.asp 
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e ~=Pennsylvania: 
http://www.portal.state.pa.us/portal/server.pt/community/Career_&_Technic 
al_Education/7335/ 


Accountability and evaluation 
Valid and reliable data systems need to be in place to gather relevant data to analyze the ef- 
fectiveness of a particular POS. The data points collected should reflect the values and im- 
portant elements for the partnership sponsoring the POS. Ideally, a longitudinal data system 


would be in place to gather and report trend data over time. 
Useful guidance included the following: 


e Guidelines for state accountability requirements; 


e Guidelines for what data to collect and how and where to submit them to the 


state; 
e Definitions of data elements and formulas; and 
e Suggestions for how to assess POS outcomes. 


New Mexico and Wisconsin have examples of monitoring and evaluation resources, as well 


as professional development resources for accountability, on their websites at: 


e New Mexico: http://www.ped.state.nm.us/CTWEB/monitoring.html 
e Wisconsin: http://systemattic.wtcsystem.edu/qrp/default.htm 
Partnerships 
Stakeholder relationships among secondary educators, postsecondary educators, workforce 
development agencies, and employers establish an ongoing forum for POS development and 


improvement. Exchanges among partners can identify economic and workforce trends that 


may direct the creation of high-demand POS based on industry needs. 
Useful guidance included the following: 


e Guides on building advisory committees and involving industry partners; 


e Sample memoranda of understanding that spell out the roles and responsibilities 


of each party (e.g., district and college); 


e Sample agendas for meetings between secondary and postsecondary partners 


and between educators and business representatives; and 


e = List of statewide workforce/industry partnerships. 
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Arizona and Kansas have posted useful examples for the development and operation of an 
effective local advisory committee made up of educators and local employers that could 


support the development of effective POS. Their resources can be located at: 


e = Arizona: http://www.ade.state.az.us/cte/info/LeadershipGuide.pdf 
e Kansas: http://www.ksde.org/Default.aspx?tabid=1660HACH 
Credit transfer agreements 
Where appropriate, credit transfer provides opportunities for secondary students to earn col- 
lege credit when they have attained an agreed-upon level of postsecondary proficiency. The 
college credits earned are entered onto transcripts at the time the student earns the credit. 


Formal agreements between secondary schools and postsecondary institutions are important 


to sustain policy-level support for such credit agreements. 
Useful guidance included the following: 


e Clear policy guidelines for funding for dual credit included in a formalized artic- 


ulation agreement; 
e §=Articulation agreement templates; and 
e Guidance and requirements for statewide articulation agreements. 


Georgia and Florida offer online examples of guides for dual enrollment, and Georgia also 


offers training videos online at: 


e = Georgia: http://www.doe.k12.ga.us/ci_cta.aspx?PageReq=CICTASeam 
e = Florida: http://www.fldoe.org/workforce/technicalassistancepapers.asp 
Teaching and learning strategies 
Instruction that explicitly shows the relationship between academic principles and technical 
application calls for new, innovative teaching and learning strategies. An interdisciplinary 


team of academic and technical educators can be encouraged to develop and adopt project- 


and problem-based teaching and learning approaches. 
Useful guidance included the following: 


e Guidelines and examples for integrating CTE and academics; and 


e  Resoutce toolkits. 
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Tennessee offers examples of how to design teaching and learning strategies that are aligned 
to industry standards for local instructors. These resources can be found at: 


http://www.state.tn.us/education/cte/ad/clupos/index.shtml. 


Technical skill assessments 
Measurement of student technical skill attainment should be explicit and assess whether the 
student’s level of technical proficiency meets what is expected by a particular industry. In 
addition to assessing a student’s technical skill attainment, assessments can also result in the 


award of an industry-recognized credential or a postsecondary certificate or degree. 
Useful guidance included the following: 


e Lists of available assessments by career area, source, and cost; and 


e Links to state licensing agencies and industry websites with certification re- 


quirements. 


Minnesota, Mississippi, and South Carolina have examples of resources for technical skill as- 


sessments on their websites at: 


e =Minnesota: http://www.cte.mnscu.edu/programs/Technical_Skill_Atta.html 
e Mississippi: https://cia.rcu.msstate.edu/Assessment/ 
e South Carolina: http://ed.sc.gov/topics/careerandtech/ 
Clarity and coherence 
The expert panel also was charged with giving an overall rating on the clarity and coherence 


of POS materials, and this rating was considered an indication of the quality of POS guid- 


ance materials. 
Examples of clear, coherent guidance included the following: 


e Easily navigable, user-friendly websites organized around CTE and POS; and 


e Accessibility of state staff for technical assistance (whom to contact for specific 


topics). 


States rated highly (3.5 or above) for overall clarity and coherence of POS guidance materials 


were California, Florida, Nebraska, Pennsylvania, Texas, Utah, and Wisconsin. 
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Summary 


In general, states have made progress in developing POS guidance aligned with the Depart- 
ment’s Framework components, even though the Framework was not released until early 
2010. No state’s guidance materials were rated very useful across the board, but many states 
had at least one component of POS guidance in which they excelled. Further, for each of the 
10 components, panel members were able to identify one or more documents considered 
exemplary for use by the field. Panelists rated seven states highly for the clarity and coher- 
ence of their POS guidance materials. 


The expert panelists found that the most useful state POS guidance provided the following: 


e State requirements for program approval or expectations for POS operations 


(e.g., accountability); 


e Funding guidelines detailing how POS can be supported (e.g., for dual enroll- 


ment or industry certifications); 
e Clear answers to FAQs, with examples; 


e Templates or samples of suggested documents (e.g., memoranda of understand- 
ing, articulation agreements, and POS), along with instructions for how to use 


them; 
e Definitions of key federal and state terms (e.g., dual credit); 


e § Crosswalks that support local subgrantees in linking rigorous technical content 


with academic and industry standards; 
e Business and industry information (local, state, and national); 


e Resource toolkits, such as PowerPoint presentations and videos related to the 
development of POS, the integration of academics and CTE, and work-based 


learning; 


e Career guidance information for students and parents and guidance counselors 


and school personnel; and 


e Contact information for state CTE staff, calendars for local technical assistance 


workshops, and conference calls for state and local directors. 


Panelists expressed concern that websites for some states were disorganized or offered 
sparse, outdated, or confusing information. Recognizing that local program staff often rely 
on the Internet for information on Perkins IV requirements, panel members judged such 
websites to be a substantial obstacle to POS design and implementation. In addition to a lack 


of useful guidance on some component areas, the panel also noted a general absence of 
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guidance or information for the postsecondary level. Materials tended to focus on the design 
and implementation of POS at the secondary level, with relatively little attention paid to the 


unique needs of postsecondary faculty and administrators. 


State POS guidance was strongest in the areas of Legislation and Policies, Course Sequences, 
College and Career Readiness Standards, and Guidance Counseling and Academic Advise- 
ment. Panelists noted, however, that many states provided information on high school 
course sequences and requirements for entry into college, but little to no information on ca- 
reer preparation specifically. That is, the “career” portion of college and career readiness was 


notably absent from state POS guidance. 


State POS guidance was weakest in the areas of Credit Transfer Agreements, Teaching and 
Learning Strategies, and Technical Skill Assessments. The expert panel suggested that states 
need to improve their guidance to locals in these areas and may need federal assistance in 
doing so. Recognizing the dynamic nature of the field, panelists also recommended that POS 
guidance and states’ approaches to implementation be reviewed again in several years to as- 


sess progress. 


Expert panelists responded positively to the Department’s Framework and concluded that 
having a guiding structure for organizing states’ design and implementation of POS would 
promote a more comprehensive, consistent, and informed response to Perkins IV. Panelists 
suggested that if states provided such guidance to local providers and shared useful re- 
soutces, it would greatly improve the development and implementation of POS nationally. 
The materials reviewed by the expert panel for this study were considered a first step in ad- 
vancing this national effort, and panelists supported circulating specific materials as an ex- 


ample of promising approaches that might benefit other states. 
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Expert Panel Rubric 


State Program of Study Guidelines 
Expert Panel Evaluation Rubric 


Overview 


The rubric you will be using to rate state Program of Study (POS) guidance to local providers is based on an emerging design frame- 
work developed by the Office of Vocational and Adult Education (OVAE), U.S. Department of Education. This framework describes 
10 core components that support the design and implementation of effective POS. Although the framework was not available to states 
at the time they initiated POS development, you will be assessing whether (and, if so, how) states are incorporating any key compo- 
nents into the guidance they offer to local providers. You will also be evaluating the clarity and coherence of the materials from the 


point of view of local providers developing and implementing POS. 


Although states will not be ranked in the final Expert Panel Review report, we will be using your ratings to describe the presence or 
absence of state guidance around each component and to assess the usefulness of the materials in supporting local providers in creat- 
ing effective POS. As you review state documents and materials, we ask that you (1) relate each of them to as few or as many of the 10 
components contained in the OVAE framework as apply, explaining why you believe each document fits the category, and (2) why you 
believe the materials may or may not be useful to local agencies. We also ask you to provide a 4-point “usefulness” rating to each doc- 
ument. There is space at the end of the rubric to list documents or materials that do not fit into any of the 10 components. Finally, we 


will ask you to assess the clarity and coherence of the materials. 


Once you have completed your review, you will participate in an expert panel webinar, along with other expert panelists and the study 
team, to discuss your state ratings. You will receive your state materials at least one week in advance of the webinar during which time 
your ratings will be reviewed, and we ask that you submit an electronic copy of your review of each state to Corinne Alfeld 


(calfeld@aed.org) at least 24 hours prior to that webinar. 
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State Program of Study Guidelines 
Expert Panel Evaluation Rubric—Continued 


Instructions 


Listed below are the 10 supporting components of POS included in the OVAE design framework, along with examples of how each 
component might be translated into state guidance. As you review each state POS document, we ask that you first consider which of 
the 10 components the item addresses, with the understanding that some documents, or even sections of documents, may apply to 
multiple components. Once you have identified the POS component(s) addressed by a guidance document, we ask that you justify 


your classification and evaluate its usefulness to local providers. 


Specifically, for each component, we ask that you provide a short description of the relevant document (or appropriate section of the 
document) (Column A) to identify it and how it addresses the component in which you have classified it (Column B). Next, describe 
your perception of the usefulness of the document in supporting local providers as they develop and implement POS (Column C) and 


assign a point value to the item using the following 4-point scale (Column D). 
1 = Not Useful 2 = Somewhat Useful 3 = Useful 4 = Very Useful 


Underneath the chart for each component, rate the overall usefulness of all of the materials you listed under that component. (If there 
was only one document, the overall rating will be the same as that of the document; if there is more than one document, you may de- 
cide to average their ratings to indicate overall usefulness.) Please add explanatory comments about your rating decisions whenever ap- 


propriate. 


Finally, in addition to addressing each of the 10 components, please also evaluate the clarity and coherence of each of the POS guid- 
ance documents on the final page of the rubric (#11). 


STATE: REVIEWER: 
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Expert Panel Rubric—Continued 


1. Legislation and Policies 


States may adopt statewide legislation or administrative policies to promote POS development and implementation at the local level. Evidence of such 
state support could include the following: 

e Legislation that formally describes the required components of POS; 

e Fiscal resources that commit state funds in support of POS development or implementation; 

e Administrative policies that describe the required components of POS at the local level; and 

e Program approval criteria that define the minimal expectations of POS for local providers to qualify for federal Perkins funding. 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add rows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


2. Partnerships 


States may promote relationships among secondary and postsecondary education, business, and other community stakeholders to support POS design, 
implementation, and maintenance. Evidence of state support of partnership development at the local level could include the following: 
e Examples of written memoranda of understanding that spell out the expectations of partnership members; and 


e Technical assistance materials to assist providers in initiating and coordinating partnerships among diverse stakeholders (e.g., secondary, post- 
secondary, business/industry, workforce agencies). 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


3. College and Career Readiness Standards 


College and career readiness standards define what students are expected to know and be able to do to enter and advance in college, postsecondary 
training, or careers. Evidence of state support for the development of such standards could include the following: 

e State-identified academic, technical, and/or work readiness standards or skills that local providers may adopt or adapt; and 

e Technical assistance materials that providers might use to identify college and career readiness standards or skills for use at the local level. 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


4. Course Sequences 


Effective POS offer a nonduplicative sequence of secondary and postsecondary courses that ensure students can transition to postsecondary education 
or training without duplicating classes or requiting remedial coursework or preparation. Evidence of state-developed materials to support local provid- 
ets in aligning course sequences could include the following: 

e Design templates/gtrids showing the academic and technical coursework in POS; 

e Technical assistance materials detailing how master schedules may be adapted to support POS delivery (e.g., block scheduling); and 


e Information on how course sequences may be structured, beginning with introductory coursework at the secondary level and progressing to 
mote occupationally specific coursework offered at the postsecondary level. 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


5. Credit Transfer Agreements 


Credit transfer agreements provide opportunities for secondary students to be awarded postsecondary credit while still enrolled in high school. Evi- 


dence of state support for local providers to award postsecondary credits could include the following: 


Technical assistance materials that describe how dual credit or dual enrollment opportunities may be structured; 

Statewide articulation agreements that formalize credit transfer opportunities across the secondary and postsecondary education systems; 
Descriptions of the expectations or requirements for teacher and faculty qualifications to teach college-level coursework; and 

Statewide tuition reimbursement agreements or resource allocation agreements. 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add rows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


6. Teaching and Learning Strategies 


Innovative and creative instructional approaches enable teachers and instructors to integrate academic and technical instruction, and students to apply 
academic and technical learning in their POS courses. Evidence of state-developed materials to support local providers in developing effective instruc- 
tional and learning strategies could include the following: 

e Examples of integrated curriculum or curricular resources (e.g., websites or state-developed curricula); 

e Technical assistance materials to assist teachers and instructors in working together to integrate coursework; 

e Provision of state-developed or identified work-based, project-based, or problem-solving curricula; and 

e Recommendations for using career and technical student organizations to support POS design or implementation. 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


7. Professional Development 


The provision of effective POS requires the sustained, intensive, and focused involvement of school and college administrators, teachers, and faculty. 
Evidence of state guidance to support professional development among local provider staff could include the following: 
e State-provided technical assistance workshops for local providers to support POS design and implementation; and 


e Training materials to assist teachers and faculty in creating effective POS (e.g., strategies for integrating academic and technical curriculum or 
aligning curriculum across education levels). 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


8. Guidance Counseling and Academic Advisement 


Guidance counseling and academic advisement help students make informed decisions about which POS to pursue. Evidence of state guidance to 


support guidance counseling and advising among local provider staff could include the following: 
e State-developed or adopted guidance and counseling planning or support documents or standards; 


e Technical assistance workshops or trainings to ensure that guidance, counseling, and advisement professionals have access to up-to-date infor- 
mation about POS offerings to aid students in their decision making; 


e Web-based tools or written resources to assist students in identifying their career interests and aptitudes; and 
e Information and resources for parents to help their children prepare for college and careers, such as workshops on college and financial aid ap- 


plications. 
Document name Why did you How useful would Rating of 
ot description classify it here? these materials be? Why? usefulness 
Pp 'y 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessary.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


9. Technical Skill Assessments 


National, state, and/or local assessments can provide ongoing information about the extent to which students are attaining the necessary knowledge 
and skills for entry into and advancement in postsecondary education, training, and careers in their chosen POS. Evidence of state guidance to support 
technical skill assessment at the local level could include the following: 

e Examples of state-adopted, industry-approved technical skill assessments for POS, where available and appropriate; 

e Technical assistance materials to assist local providers in developing or selecting technical skill assessments; and 


e Evidence that states may award secondary credit, postsecondary credit, or a special designation on a student’s high school diploma for those 
who pass assessments. 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


10. Accountability and Evaluation 


Systems and strategies to gather quantitative and qualitative data on both local POS components and student outcomes are crucial for ongoing state 
efforts to support development and implementation of POS at the local level. Evidence of state guidance to support local providers in assessing and 
addressing the outcomes of POS adoption could include the following: 

e Written guidance on criteria for identifying POS participants and tracking their outcomes; 


e State strategies for conducting follow-up of students completing the secondary component of POS; and 
e State tracking of POS participants. 


Document name Why did you How useful would Rating of 
(or description) classify it here? these materials be? Why? usefulness 
(Column A) (Column B) (Column C) (Column D) 


(Add tows as necessaty.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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Expert Panel Rubric—Continued 


11. Other 


Other observations about the documents which do not fit into the 10 previous components. 


Document name Why did you How useful would 
(or description) classify it here? these materials be? Why? 
(Column A) (Column B) (Column C) 


(Add tows as necessaty.) 


Comments: 


B-29 


Rating of 
usefulness 
(Column D) 
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Expert Panel Rubric—Continued 


12. Overall Clarity and Coherence 


Effective state guidance to support local POS development and implementation hinges on whether local providers can access and use the information 


states provide. Evidence that state written guidance is accessible and designed for local use could include the following: 
e References to state websites where POS information is posted; 


e Identification of state technical assistance phone numbers or state staff members who can answer questions relating to POS development and 
adoption; and 
e Rubrics that organize materials or other signs that guidance is clearly written, logically organized, and easily understandable. 


Document name 


(or description) Comments on the clarity, accessibility, and coherence of the materials 


(Add tows as necessary.) 


Overall rating of usefulness of state guidance to local providers on this component 


Comments: 
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E-mailed Letters to States 
Requesting POS Guidance Materials 


Initial E-mailed Letter to State Directors 


Dear [Name of State Director], 


Thank you for your continued support in helping us to collect information for use in this 


study. 


Upon reviewing the survey you recently completed, we have determined that we are missing 
important information about the guidance your office provided to assist local administrators 
in designing and implementing POS. Since we will be convening a panel of experts to review 
the guidance and support provided by states, it is critical that we have information about 
what your state has offered to the field. The panel review will not be evaluative nor will it be 
used to make comparisons among states; rather, expert panelist findings will be used to ob- 
tain a better understanding of states’ POS design process, and to craft recommendations that 


will support all states in creating and/or refining POS for career and technical education. 


Could you please provide us with contact information for the staff member within your state 
who can provide us with copies of any POS guidance and materials that your state has 
shared with local agencies? Information should be sent to Corinne Alfeld (calfeld@aed.org). 
If you have any questions, please contact me at (503) 963-3757. 


Many thanks, 


Steve Klein, Study Director 
Corinne Alfeld, Expert Panel Lead 
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Follow-Up E-mail Letter to Designated State Staff 


Dear : 
I received your contact information from [Name of State Director]. 


We are compiling materials from each state to help us understand the state agency’s role in 
helping local providers develop and implement Programs of Study (POS). We have assem- 
bled an expert panel to review these materials with the goal of identifying promising exam- 


ples and strategies that can inform other states and local providers. 


Iam hoping that you will be able to provide us with materials and information that your 
state has made available to support local providers in creating POS. We are only interested 
in existing materials that you have already shared with your substate subgrantees 


since the reauthorization of the Act. 


Specifically, we would like any or all of the following types of documents, in either electronic 


ot hard copy, or through a URL: 


e Statewide definition of POS 

e State guidelines and/or templates (e.g., forms and instructions, model standards, assess- 
ments, articulation agreements, suggested course sequences, guides for aligning academic 
and CTE standards and secondary and postsecondary curriculum, and any administrative 
regulations released by state career and technical agency staff to local providers) 

e Statewide legislation and guidance on POS, articulation agreements, and/or dual enrollment 

e Statewide guidance or requirements regarding college and career readiness stand- 
atds/assessments and industry certifications 

e State’s approval process for POS 

e Materials distributed at state-provided professional development training on POS 

e Guidance and counseling tools developed to support POS, such as Individual Graduation 
Plan templates and college and career planning resources 


e Other materials 


We may have already collected some of these materials via your state ditector’s sur- 
vey and seatches of your state’s website. 


Please feel free to call me at (202) 884-8622 if you would like to discuss this request. 


Thank you! 
Corinne Alfeld, Expert Panel Lead 
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B-33 


Information on POS Collected from State Websites 


(Example: Alabama) 


Link to main CTE page on state website 


Statewide definition of POS 


POS offered 


Link(s) to guidelines and template(s) developed by state to 
help local providers develop POS (e.g., forms and instruc- 
tions, guides for aligning academic and CTE standards and 
secondary and postsecondary curriculum, and any administra- 
tive regulations released by state career and technical agency 
staff to local providers) 


Link(s) to guidelines/instructions and approval 
form(s)/criteria for local providers to submit POS for state 


approval 


Other relevant links 


http://www.alsde.edu/html/sections/section_detail.asp?secti 
on=52&footer=sections 


POS not mentioned. Career Clusters defined as “Career clus- 
ters in Alabama include courses that identify academic and 
technical knowledge and skills needed for students to pursue 
a wide range of career opportunities. Courses provide rigor 
and relevance for students by linking school-based learning 
with career-related experiences. Career clusters provide the 
framework for what students need to know and be able to do 


for success in the twenty-first century.” 


CTE areas listed on the website include Agriscience, Law, 
Public Safety and Security; Commerce and Information 
Technology; Family, Human Services, Hospitality and Tour- 
ism, and Education; Health Science; and Technical Systems 
Education. There is a section on Tech Prep with a link to the 
Statewide Articulation Memoranda of Understanding listing 
all courses approved for statewide articulation. Career Clus- 
ters, Pathways, and Courses are listed on the website. The 
website lists 16 National Career Clusters. 


Advisory Committee Handbook and Handbook for Local 
CTE partnering are available as Word documents on main 
CTE website. (Note: neither document mentions “program 
of study.”) 


Not mentioned 


 http://www.alsde.edu 
(Sections > Curriculum & Instruction > Publications - 
Courses of Study Other > Combined courses for all 300 
developed) 

 http://www.alcareertech.org/ 
(AL Department of Education CTE website, most infor- 
mation outdated) 

e http://www.alcareerinfo.org 
(Resources > Education Corner > Plans of Instruction) 


unit plans 
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State POS Guidance Examples Recommended by the 
Expert Panel as National Models 


State Materials Recommended as National Models (listed in 
alphabetical order) 


State Useful guidance materials 


California e CTE framework document (http://www.cde.ca.gov/ci/ct/sf/documents/cteframework.pdf); 

e Standards validated by industry and crosswalked to connected curriculum framework with many resources 
(http://www.cde.ca.gov/ci/ct/sf/documents/ctestandards.pdf); 

© State career pathways websites with information on articulation agreements 
(http://www.statewidepathways.org/); 

© Career guidance (The Real Game and Road Trip Nation—part of California Career Resource Network) 
(http://www.californiacareers.info/); 

e Work experience education program information (http://www.cde.ca.gov/ci/ct/we/); 

e Data systems 


Florida e Technical assessment FAQs; 
© Perkins implementation guide (http://www. fldoe.org/workforce/perkins/pdf/2010-2011RFA.pdf); 
e White papers (technical assistance papers?) http://www.fldoe.org/workforce/technicalassistancepapers.asp) 


New Mexico e Monitoring documents (http://www.ped.state.nm.us/CTWEB/monitoring.html) 
Pennsylvania © Good upfront piece for navigating website 
(http://www.pottal.state.pa.us/portal/setver.pt/community /Career_&_Technical_Education/7335/); 


e “Tlluminate” online communication tool 


South Carolina e Entite website could be a model (addresses multiple needs) (http://ed.sc.gov/topics/careerandtech/); “Paving 
Pathways to Success”; cost and funding options for technical assessments; data system 


Texas e Website materials better than others (http://ritter.tea.state.tx.us/cte/index.html); PD videos 
(http://cte.tamucc.edu/la/online_library.shtml); AchieveTX.org (http://www.achievetexas.org/) 


Other Notable State Materials (listed in alphabetical order) 


Arkansas e PD sample technical assistance letter; 
e PowerPoints on how to improve programs and POS 
(http://ace.arkansas.gov/CareerandTechEducation/presentations.htm) 


Arizona e Leadership guide for creating local advisory committees 
(http://www.ade.state.az.us/cte/info/LeadetshipGuide.pdf) 


Colorado © Career guidance (http://www.coloradostateplan.com/Counselors.htm); 
e Overall clarity of website materials (http://www.coloradostateplan.com/index.htm) 


Georgia © Guide to dual enrollment (http://www.doe.k12.ga.us/ci_cta.aspx? PageReq=CICTASeam); 

e Training videos (http://www.doe.k12.ga.us/ci_cta.aspx) 
Hawaii e Glossary of terms (career pathways, career clusters, POS) (http://www.hawaii.edu/cte/pathways/glossary.html) 
Illinois e POS guide (though no postsecondary component) 


(http://occtLillinois.edu/files/Projects/perkins /POSguide.pdf) 
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State 


Indiana 


Kansas 


Minnesota 


Missouri 


Mississippi 


North Carolina 


Nebraska 


New Jersey 


New York 


Ohio 


Oklahoma 


‘Tennessee 


Utah 


Wisconsin 


Wyoming 


Useful guidance materials 


Course sequence template (http://www.doe.in.gov/pathways/); 
State policy on credit transfer (http://www.transferin.net/index.aspx); 
FAQs on dual credit/dual enrollment (http://www.doe.in.gov/core40/pdf/Dual_Credit_QA.pdf) 


Guide for advisory committees (though MOA not included); 
Links to PDFs (http://www.ksde.org/Default.aspx?tabid=1660#ACH); 
FAQs are good; video and podcast training (good site but not POS specific) —same link. 


PDF files; 
Frameworks (link to example framework) (http://www.cte.mnscu.edu/programs/index.html); 
Technical assessments (http://www.cte.mnscu.edu/programs/Technical_Skill_Atta.html) 


Good website materials, definition of POS (http://dese.mo.gov/diveareered/perkins_iv_pos.htm); 
Curticulum policies; PDFs; 
Multimedia offerings 


Assessment center (https://cia.rcu.msstate.edu/Assessment/) 


Course of study (http://www.ncpublicschools.org/cte/); 
Careers magazine; briefing papers (http://www.ncpublicschools.org/cte/briefing/) 


Good presentation of materials; technical skill assessments; 
POS information related to curriculum standards (http://www.nde.state.ne.us/ NCE/) 


Toolbox (career and college readiness) (http://www.state.nj.us/education/voc/toolbox.htm); 
POS reapproval application (http://www.state.nj.us/education/voc/occprapp.htm); 
Work-based learning; Structured Learning Experiences http://www.state.nj.us/education/voc/sle/) 


Career development and occupational resource guide = living document 
(https://www.nycareerzone.org/cz/index.jsp) 


Teacher preparation; state administrative rules; 

CTE standards 
(http://education.chio.gov/GD/Templates/Pages/ODE/ODEPrimary.aspx?Page=2&TopicID=1752&Topic 
RelationID=2) 


Oklahoma skills standards (done by a curriculum center) 
(http://www.okcareertech.org/testing/skills_stands.htm) 


PowerPoints for locals (http://www.state.tn.us/education/cte/index.shtml); 

CTE website (with links to PPTs); 

Detailed lesson plans (searchable); 

Standards alignment to POS for technical exams 
(http://www.state.tn.us/education/cte/ad/clupos/index.shtml); state POS website; grants for dual credit 
(http://www.tn.gov/CollegePays/mon_college/grants.htm); state data system (E-TIGER) 
(http://www.state.tn.us/education/cte/ad/tiger/index.shtml) 


Good site, easy to navigate; program approval by cluster (http://www.schools.utah.gov/cte/pathways.html) 
Summary of POS used in school districts and news and resources (http://dpi.wi.gov/cte/index.html) 


Guides related to careers and clusters within state (http://www.k12.wy.us/ICS/Cluster_Guides.asp) 
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Appendix C. Cross Case Report 


Introduction 


This appendix summarizes a series of case studies examining how, why, and to what extent 
statutory changes led to desired changes in CTE programs. The case studies, involving six 
states and three communities within each state, are intended to provide a deeper understand- 


ing of Perkins IV implementation at state and local levels. 


National experts recommended the six participating states based on their size, geography, 
and organizational structure. Research teams visited the case study sites to collect qualitative 
data about their experiences in developing accountability systems, financial systems, and 
programs of study (POS) in response to Perkins IV. Interview protocols were used to struc- 
ture the information collected through the case studies and to promote consistent, high- 


quality data collection across the case study sites. 


Methodology 


In five of the six states, the eligible agency was the state department of education; in the 
sixth state, an agency charged with coordinating workforce development and education ac- 
tivities was the recipient of Perkins funds. Local communities were selected by using a table 
of random numbers to identify local education agencies (LEAs) from the state’s Common 
Core of Data file, collected by the Department’s National Center for Education Statistics. 
For case study purposes, a community was defined as a school district and/or area CTE cen- 


ter and its postsecondary partner(s). 


Each case study began with in-depth interviews with state secondary and postsecondary 
CTE directors to gather perspectives on their state’s progress in implementing Perkins IV 
and challenges they had encountered. In addition, researchers interviewed state-level staff re- 
sponsible for the accountability and finance provisions of Perkins IV as well as those respon- 
sible for the design and implementation of POS. Information obtained at the state level 
provided both context and a knowledge base for local site visits. In the local communities, 
researchers met with CTE directors,! secondary and postsecondary administrators and facul- 


ty, and business partners. 


'Tn most LEAs, the CTE director has responsibility for implementation of Perkins-funded activities; 
in smaller districts, the CTE director may have other administrative or teaching responsibilities. In 
THEs), the lead staff person for Perkins is typically a dean, a department head of a faculty member; 
typically there is not an individual with the CTE director title. 
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All respondents were assured anonymity. Researchers told each respondent that no state, lo- 
cality, or individual would be identified in the research or findings and that the information 

they provided would not be attributed directly to them by name, position, or location. Some 
of the information and quotations in this appendix have been edited slightly to preserve con- 


fidentiality. 


To promote consistent, high-quality data collection across the case study sites, the research 
team created a common set of protocols in consultation with Department staff. These pro- 
tocols were tailored to the roles and responsibilities of state and local program administra- 
tors and staff. (Copies of these protocols are provided in Attachment A.) In addition, all case 
study staff participated in a full-day training session that included an in-depth review of the 


protocols, qualitative methodology, and expectations for note-taking and report writing. 


Two- or three-person teams conducted all of the site visits between January and May 2010. 
A senior staff member with extensive case study experience served as the lead for all visits to 
a state and the communities within it. This individual worked closely with state and local 
CTE administrators to develop site visit agendas and arrange interviews with key personnel. 
Other members of the team assisted with interviews and assumed primary responsibility for 
note-taking. To ensure that researchers had thorough notes for analyses, most case study in- 


terviews were recorded and professionally transcribed. 


Findings from approximately 30 interviews at the state level and more than 100 interviews at 
the local level revealed a number of patterns and common themes. Although it is not possi- 
ble to generalize case study findings to the larger universe of states and eligible recipients en- 
gaged in the design and delivery of CTE programs and activities, these findings present a 
clear snapshot of Perkins financing, accountability systems, and POS in the 2009-10 school 


year. 


CTE in the Context of Secondary Education Reform 


In several of the case study states, both state and local administrators reported attempts to 
enhance CTE as part of broader education reforms at the secondary level—some of which 
were driven by the requirements of the E/ementary and Secondary Education Act (ESEA), while 
others were part of efforts to raise students’ academic performance and reduce dropout 
rates. A number of state and local secondary CTE administrators described how CTE was 
an integral component of these reform efforts. At the same time, many of these administra- 
tors said that the reform efforts have been undermined by significant cuts in state funding, 
which have reduced staffing and resources. In contrast, few postsecondary respondents de- 


sctibed ways in which CTE programs were part of more extensive reform strategies. 
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CTE and State Reform Efforts 


Reform efforts that encompass CTE varied across the states. Secondary reform strategies 

that highlight CTE included requirements for career exploration and advising at the middle 
school level; incorporation of CTE into the state’s system for grading school performance; 
patticipation of CTE practitioners in the state’s high school redesign committee; consolida- 
tion of CTE and academic units within the state department of education; and adoption of 


cateet-ready standards. 


One state recently enacted statutes encouraging students to begin thinking about career- 
oriented studies as early as middle school. As a result, middle school students are expected to 
use an online advising system to create an academic and career plan from sixth grade 
through postsecondary education and are required to take at least one career course before 
promotion to high school. Another important change in this state was integrating CTE into 
the state system for grading school performance. Previously, schools were graded solely on 
their standardized test scores. Under the new policy, the grading system was expanded to in- 
clude the number of industry certifications earned by students in each school—a measure 
that state staff said made CTE and workforce development more visible. In addition, state 
administrators reported efforts to forge stronger links between schools and employers by re- 
quiring local CTE programs to coordinate with local business and industry and focus on in- 


troducing students to high-wage, high-demand jobs. 


The CTE director in another state described the participation of a CTE staff member in the 
state department of education’s internal high school redesign committee. “We felt like we 
played an instrumental role in driving high school redesign with the new graduation re- 
quirements,” he said. “As we redevelop curriculum, we’ll use the new academic standards 
that have been implemented and highlight those in each one of the program areas where it’s 


appropriate.” 


In a third state, the CTE director outlined ambitious plans to create integrated school sup- 
port units at two levels, P—8 and 9th grade through postsecondary education, within the state 
department of education. He had been tapped to lead the latter division, giving him over- 
sight of high schools, CTE, and adult education and longer range goals of revising govern- 
ance policies and funding formulas to link elementary, middle, and high schools with area 
CTE centers through a regional allocation. He also cited efforts to strengthen relationships 
with other departments, a move he described as “really getting out of our agency silos.” As 
an example, he cited an analysis of the agriculture and natural resources sectors in the state 


that was funded by the state department of education. 


Administrators in another state reported developing career-ready standards that encompass 
“critical thinking, teamwork, responsibility, and all of those types of things that you don’t get 


a measurement of on the pure academic side of the house.” They said they used some Per- 
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kins funds to bring together more than 80 leaders from education and industry to discuss 
these standards and noted that they “purposely branched out beyond just CTE.” State ad- 
ministrators said that LEAs have broad discretion in setting course requirements, as long as 
they meet state academic standards. They indicated that this flexibility allows districts to de- 
velop courses such as vocational agriculture that meet state math requirements and are as ac- 


ademically challenging as an algebra I course. 


In some states, CTE played a central role in policymakers’ efforts to keep struggling students 
interested in school and provide them with valuable workforce skills. One state CTE direc- 
tor suggested that CTE programs are important in overall efforts to improve high schools by 
helping struggling students understand the relevance of various academic topics. “We see 
CTE in a broader scope of high school improvement,” he remarked. While many students 
are comfortable with lessons offered through “abstract and conceptual, typical traditional” 
strategies, he asserted that those approaches risk ignoring the needs of the “middle majority” 


of students “who learn more from experiential learning.” 
g 


His views were echoed by his counterparts in other states, one of whom observed, “All of 
the research shows that you’ve got to get a student engaged. It may be in band, it may be in 
art, or it may be in CTE, but those students [who] are engaged graduate at higher rates than 
those who are just window shopping for courses.” Similarly, officials in another state said 
that they have paid increasing attention to improving high school completion rates in recent 
years and suggested that effective CTE classes can encourage students to stay in school. 
While CTE is not the only dropout prevention strategy, it is “very important to keeping kids 
in school, keeping them engaged and excited about learning,” one state official explained. In 
this state and others, officials also expressed concerns that students had fewer opportunities 
to take CTE courses and other electives as a result of ESEA and increased high school 


graduation requirements. 


In at least two states, administrators reported that budget cuts had led to reduced CTE staff- 
ing at the state level and other changes to CTE program delivery. In one state, budget cuts 
resulted in the conversion of the state’s designated CTE funding stream into flexible funding 
that allows districts to use money that once paid for CTE and other programs and services. 
Officials in this state said they expected this flexible funding arrangement to continue to put 


pressure on CTE budgets in the years ahead. 


CTE and Local Reform 
Administrators in several LEAs noted that CTE played a significant role in school and dis- 
trict reform efforts. Some suggested that CTE helped forge stronger ties with postsecondary 
institutions and led to better alignment of academic programs with postsecondary expecta- 
tions, citing dual enrollment policies designed to boost students’ access to academically chal- 


lenging course content. Others explained how districts have made creating new and more 
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academically rigorous CTE programs a core element of efforts to redesign struggling 


schools. 


One local CTE director cited CTE as a major part of the LEA’s efforts to redesign 14 low- 
performing schools and noted district leaders’ conviction that CTE programs meet students’ 
interests and capitalize on their analytical and creative skills. Administrators in this LEA 
sought to improve the overall quality of CTE courses. One official noted that CTE courses 
are no longer regarded as a “dumping ground,” but rather as a “proving ground.” As part of 
their effort to improve CTE quality, staff in the LEA created honors-level CTE courses and 
integrated CTE courses with other disciplines, such as foreign languages. For instance, the 


LEA was offering a “business Spanish” class. 


As pressure to reduce dropout rates intensifies, several LEA officials said that they regarded 
CTE as an important vehicle for keeping students interested in school and helping them un- 
derstand its relevance to their lives. One local CTE director recalled considerable debate in 
his district about whether teachers should focus on preparing students for college, the work- 
force, or both. He reported that LEA administrators ultimately recognized the importance of 
CTE courses in providing students with the academic foundation necessary to choose either 
a postsecondary or career path. In addition, he noted a focus on ensuring that CTE courses 
are more than just “fun electives” and explained that district officials have become increas- 
ingly determined to create “deeper, stronger links” to postsecondary education and support 
students who learn in nontraditional ways. An administrator in another district reported data 


on the connection between CTE and high school completion: 


We have the data to prove that career and technical education students are 
graduating at a higher percentage rate. We’re retaining those students... . 
Today’s students want to know “Why do I need to learn this?” I can re- 
member asking that question in high school myself. And I remember my al- 
gebra teacher saying, “Oh, you'll use it someday. Don’t worry about it.” 
Well, we’re actually showing students how to use algebra, and they’re apply- 


ing the concepts. They use them, and so they retain them. 


In another LEA, officials reported that they were organizing CTE classes and other elective 
courses around unified themes as part of an ES'F.A-mandated school restructuring initiative 
focused on creating academies or small learning communities. Beginning in the 2010-11 
school year, all students in the district will select a small learning community when they 
move from middle to high school. Each small learning community will contain several CTE 
pathways and formal POS. The local CTE director said the small learning communities will 
be the central pathway through which students will be exposed to CTE in the future. She al- 
so noted that CTE courses are being embedded in small learning communities within 


schools in different ways. In some schools, they are offered as electives that are not ap- 
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proved for credit for admission to state universities; in other schools, CTE courses do meet 
this state standard. One school offered a sequence of three CTE courses, with the most ad- 
vanced course eligible for college credit. District officials said that they intended to review 

the different structure of CTE courses within schools’ small learning communities in the fu- 


ture. 


Several K-12 and postsecondary officials said that CTE is critical to efforts to forge stronger 
links between high schools and postsecondary education, in some cases through dual- 
enrollment agreements between LEAs and institutions of higher education (IHEs). In one 
LEA, increased interest in promoting college access and workforce preparation had generat- 
ed support for dual enrollment and the establishment of a “regional collegiate high school.” 
A regional administrator in another state noted that Perkins IV had encouraged much closer 
working relationships between K—12 systems and colleges than had existed in the past, when 
those links were more “haphazard.” Today, schools have “colleges sitting at the table with 


us, talking to us as partners,” he remarked. 


CTE and Postsecondary Reform 
In several states, IHEs were building on existing reform efforts in implementing Perkins IV, 
especially in developing connections with secondary partners. As one state postsecondary 
administrator explained, “There was a lot of precursor work [before Perkins IV]. . . the sys- 
tem spent a heck of a lot of energy well before Perkins IV around career pathways, career lat- 
tices that came out of the career clusters work, the 16 career clusters.”’ Another 
postsecondary administrator described her institution’s intensive efforts to develop dual en- 
rollment opportunities for secondary students and why they chose to concentrate those ef- 
forts in area CTE centers: “We very intentionally have dual enrollment situated in the [area 
CTE centers] because we believe that we can support the middle flyers, a lot of whom are at 
the [area CTE centers], who don’t think of college as an option. [T]his is an opportunity for 
them to see a future beyond simply a high school degree.” This state’s community college 
system had established a staff position dedicated solely to forging connections with second- 
ary partners. This staff person explained how secondary-postsecondary partnerships in the 
state had been built over time: “There’s a lot of agreement [between secondary and postsec- 
ondary partners] because we’ [ve] already done a lot of the leg work. ... We have about eight 
years of an exceptionally strong partnership between the community college and the [area 
CTE centers].” 


Academic Credit for Secondary CTE Coursework 
Policies related to awarding academic credit (1.e., required or elective credits in specific sub- 
jects) for CTE courses varied considerably across the case study states. The decision about 
awatding academic credit for coursework completed in a CTE subject area or class was gen- 


erally left to LEA officials in most of the states. However, the latitude allowed LEAs by state 
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officials varied. On one end of the spectrum, state university system officials had identified 
7,500 individual CTE courses that met the system’s admissions requirements, but local 
school boards still had to decide whether students received academic credit for any particular 
CTE course. The CTE director in another state said that there were a number of contextual 
academic and CTE courses that could be substituted for academic courses. In a third state, 
the CTE director said that decisions about academic credit are made at the local level. “It’s 
not state-controlled or state-approved,” he noted. “There are local schools that offer science 


ot whatever credit for CTE courses.” 


Administrators in another state described an alternative credit pilot program launched in 
2008 to allow students enrolled in industry-certified CTE courses to be awarded core credits 
in biology, geometry, or algebra. Examples of CTE courses qualifying for academic credit in 
this state included an agricultural science foundation course that meets a general science te- 
quirement and a health science course that leads to credit in anatomy/physiology. Officials 
also noted that some CTE courses can meet the state’s graduation requirements for fine atts. 
CTE administrators were optimistic that more CTE courses would qualify for academic 
credit in the future, because the state’s application for federal “Race to the Top” funding 
would require districts to offer CTE courses in science, technology, engineering, and math 


that count for academic credit and offer industry certification. 


However, the CTE director in another state was less optimistic about the future of academic 
credit embedded in CTE courses. He said that embedded academic credits are recorded at 
area CTE centers and then on transcripts at the students’ home high schools but that some 
high schools refuse to recognize these credits. “I started saying in 2006 that you need to as- 
sume that embedded academic credits will go away,” he recalled. Describing these credits as 
a “threatened species” if not validated through an assessment process, he questioned wheth- 
ert embedded academic courses offered at area CTE centers are “valid and reliable methods 
of teaching students the same level of academics that a student not attending the regional 
center would get.” A college administrator in the same state reported difficulties in determin- 
ing what levels of academic courses were embedded in CTE classes. She said it is “really dif- 
ficult for a college to figure out what a student’s got for math when the student’s math credit 


is embedded in his/her automotive course.” 


Implementing Perkins IV Funding Provisions 


This section of the appendix provides a summary of the major funding provisions in Perkins 
IV and describes how they are being implemented in case study states and communities. 
Particular emphasis is placed on the distribution and use of funds within states, LEAs, and 
IHEs. The section concludes with a summary of recommendations offered by respondents 


for changes to funding provisions in future Perkins reauthorizations. 
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Major Funding Provisions in Perkins IV 


Statutory provisions pertaining to the distribution and uses of funds remained substantially 
consistent from Perkins III to Perkins IV. There were no changes to the formula used to dis- 


tribute money to states or to the general requirements for allocations within states. 


Perkins fands are distributed to states using a formula based on the percentages of residents 
in each state who fall within different age groups. Funds are awarded under two different 
sections of the Act—Title I, which is also known as the basic state grant, and Title II, Tech 
Prep Education. Perhaps the most significant change in Perkins IV was a new provision that 
gave states the option to merge some or all of their Tech Prep allocations with the funding 


they receive under Title I of the legislation. 


As with its predecessor, Perkins IV specifies that states are required to allocate at least 85 
percent of their Perkins Title I funds to LEAs and IHEs, although it gives states the discre- 
tion to determine how the funds are split between the secondary and postsecondary levels. 
The legislation also prescribes the formulas for fund allocations within states. At the second- 
ary level, the formula considers the number of individuals aged 5-17 within the LEA and the 
number of individuals in this age group who are below the poverty line. The postsecondary 
formula includes the percentage of students who receive Pell Grants or financial assistance 
from the Bureau of Indian Affairs. In addition, the Act identifies thresholds for minimum 
grant allocations—$15,000 at the secondary level and $50,000 at the postsecondary level— 
and requires LEAs and IHEs that do not qualify for the minimum allocation to join consor- 


tia. 


Under Perkins LIT, state Perkins administrators were allowed to reserve up to 10 percent of the 
funds to be spent at the local level (or up to 8.5 percent of the total Perkins Title I allocation) 
for distribution to local subgrantees in areas with high numbers of CTE students, rural areas, 
and areas negatively affected by changes to within-state formulas. Perkins IV allows state of- 
ficials to target these funds to areas with high percentages of CTE students, areas with large 
numbers of CTE students, and rural areas. (The language pertaining to within-state formulas 


was eliminated because the formulas were not changed.) 


Perkins IV allows states to reserve up to 10 percent of their Title I allotments for state lead- 
ership activities. The legislation identifies 9 required and 17 permissible uses of leadership 
funds. Required activities include assessing CTE programs; promoting and improving the 
use of technology in CTE programs; supporting professional development of teach- 
ers and other staff; providing preparation in “nontraditional” fields that expose stu- 
dents to high-wage, high-skill occupations; and providing technical assistance to local 


schools and districts. Permissible activities include strengthening guidance and career 
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counseling; establishing articulation agreements between secondary and postsecond- 


ary CTE institutions; and supporting partnerships between schools and businesses. 


As with the previous legislation, Perkins IV limits the amount of Perkins funds that can be 
used for administrative expenses at the state level to $250,000 or no more than 5 percent of 
the state’s Title 1 allotment (whichever amount is greater). States are also required to provide 


a one-to-one match for administrative funds from nonfederal sources. 


Key Findings 


All of the case study states and most of the LEAs and IHEs provided additional funding for 
CTE, either through categorical grants or general budget expenditures for teachers, adminis- 
trators, facilities, and classroom materials. Nonetheless, almost all respondents indicated that 
the need for Perkins funds exceeded the dollars available. State CTE administrators reported 
that administrative funds, which were typically used to cover staff, data collection and analy- 
sis, and travel, were inadequate especially with the new requirements of Perkins IV. They also 
noted that the amount designated for leadership funds was insufficient to cover the addi- 
tional required and permissible uses. At the local level, most practitioners were satisfied with 
the process by which Perkins funds were allocated, but almost all said that funding for CTE 
was insufficient. Key findings related to the implementation of Perkins funding provisions 


ate summarized below: 


e Four of the six states merged Tech Prep funds with their basic Perkins grants. 
There was little agreement about the decision to merge Perkins funds, postsec- 
ondary administrators generally agreed that they should be kept separate, while 


secondary administrators generally favored consolidation. 


e Across the states, decisions about the secondary and postsecondary funding 
split were based on factors such as data analyses, competitive grant processes, 
the number of eligible providers, and comparative enrollments; on average, 58 


percent of funding went to LEAs and 42 percent went to IHEs. 


e Many LEAs and some IHEs formed consortia to meet the minimum allocation 
threshold. 


e Three states provided financial support for dual enrollment through different 


funding strategies. 


e Four states took advantage of the reserve option to enhance flexibility, increase 


funding for rural communities, and support innovation and technical assistance. 
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Perkins IV Funds to States 


There was broad agreement in the states that Perkins funding is crucial to the operation of 
state and local programs, and many state and local officials said they did not believe there 


was adequate federal support for CTE. 


Most sites finance CTE with both state and local funds, which typically are used to support 
two of the biggest budget items in the educational system—teachers and facilities. One state 
supported CTE through state basic education funding. “We have a considerable investment 
in state money that goes into secondary and postsecondary CTE,” the state director of work- 
force education explained. “In fact, the postsecondary has really grown to respond to issues 
of capacity and providing a trained workforce that business and labor need.” In another 
state, support for CTE has been more problematic. As a local CTE director noted, “The 
budget here is a little bit over $3 million, but $500,000 of that is for debt reduction, because 
when the state authorized us to build this facility, they said they were going to pay for it. But 
they haven’t paid for it yet. There’s still over $9 million of outstanding debt that we have to 
pay the interest on that the state hasn’t paid the capital on yet.” 


States may choose to consolidate all or part of their Tech Prep grants with their basic state 
grants. Four of the six states visited merged Tech Prep and basic grant funds, and two kept a 
separate Tech Prep funding stream. There was little agreement within the states on merging 
funding streams or keeping them separate: postsecondary administrators generally agreed 
that they should be kept separate, while secondary administrators generally favored consoli- 
dation. In one of the four states that consolidated its funds, the process played out over sev- 
eral years: funding streams were kept separate in 2007-08; funds began to be integrated in 
2008-09; and then funds were merged in 2009-10. 


Local perspectives on the merger of Tech Prep funds often paralleled those found at the 
state level. Postsecondary administrators and instructors tended to oppose a merger, while 
most secondary administrators favored consolidation. Across the six states, the pattern ap- 
peared to rest on the perception of where additional money would be placed, with postsec- 
ondary representatives favoring separate Tech Prep funding that typically flowed through 
IHEs and secondary officials favoring consolidation because funds might be spread between 


the postsecondary and secondary levels. 


Splitting Funds between Secondary and Postsecondary Education 


States are required to provide support for CTE programs at both the secondary and post- 
secondary levels. States have the flexibility to determine the proportion of funds that each 
education level will receive, and the six states reported widely differing approaches to sharing 


Perkins IV Basic Grant funds between the secondary and postsecondary levels (Exhibit C.1). 
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Exhibit C.1. 
Split of Perkins IV funds within case study states: 2009-10 
State Secondary Postsecondary 
A 55 percent 45 percent 
B 44 percent 56 percent 
co) Beret | 2perent 
, es |e stperent = | 9 petcemt 
EB | thpercent =| percent 
; rr ae aSperent | 1S percent 
“Sixetate average = | SBpercent =| AR percent 
“DoW national average = |B percent, =| ST percent 


On average across the six states, 58 percent of basic grant funding went to LEAs and 42 
percent to IHEs, although this breakdown varied greatly from state to state. When splitting 
the funds, two states provided proportionally much more funding to secondary education, 
with 74/26 and 85/15 splits, respectively, while another took a very different approach, de- 
voting 62 percent to postsecondary and 38 percent to secondary, a breakdown nearly the re- 
verse of the national average. If anything, the case study states seemed to provide a higher 
proportion of funding to IHEs than the national average. Two states allocated more than 
half of their Perkins IV funding to IHEs, while two other states split funds almost evenly be- 
tween the two levels. 


State officials offered different rationales in describing their policies for dividing funding be- 
tween secondary and postsecondary education. Officials in the state with the 85/15 split be- 
tween the secondary and postsecondary levels said that breakdown was necessary to 
continue using a competitive grant process. IHEs in that state, however, also receive addi- 
tional support through state leadership funds and through a special grant to cover tuition for 


dual enrollment students. 


In another state, the funding split is based on postsecondary data and was in place before 
Perkins IV. In noting the almost equal split between the secondary and postsecondary levels, 
a state administrator put it this way: “Federal dollars are not considered to be the driving 
force behind CTE and workforce development, due to the level of investment at the state 
level, so pushback on the split decision has been minimal.” He continued, “The increase in 
the postsecondary allocation was due to enrollment increases and value-added data analyses, 
which made the case for increasing the postsecondary funding pot.” The division providing 


more funds to postsecondary education in one state is based on comparative enrollments in 
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CTE at the two levels. In another state, the split is based on the number of eligible provid- 
ers, resulting in 74 percent of the funds going to LEAs and 24 percent to IHEs. 


After making secondary and postsecondary allocations, one state reserved 10 percent of each 
allocation for a nonprofit organization focused on supporting innovative partnership pro- 
grams across the state. This approach provided greater flexibility for the state to allocate Per- 
kins funds and support local partnerships between secondary and postsecondary education, 


officials in this state explained. 


Another state established a working group to ensure that the split between secondary and 
postsecondary education was equitable and fair, based on full-time-equivalent (FTE) student 
enrollment and relative need. That split can fluctuate annually, which affects the formula. 
Another state has changed its rules on forming consortia based on Perkins IV provisions. 
Under Perkins IV, consortia are less able to “game the system” by providing support for dis- 


tricts within the consortia. 


Changes to Funding Consortia under Perkins IV: 
An Example 


A state-level administrator explained how Perkins IV had prompted the state to change its own 
rules for funding consortia, with implications for delivering high-quality CTE. 


This state previously had allowed schools to create their own consortia for Perkins funding. 
Some consortia then returned the funds to each district to enable the district to buy the equip- 
ment it needed. Under Perkins IV, however, school districts can only form a consortium with a 
regional education entity or with a community college. The result is that federal funds can only 
be invested in developing high-quality CTE, rather than serving as a revenue stream to pur- 
chase equipment and other resources. 


Allocating Perkins Funds to LEAs and IHEs 


States have implemented formal application processes to determine allocations to LEAs and 
IHEs. Some states showed large disparities in the amount of funds allocated to secondary 
and postsecondary education. These differences—even if explained by federal and state poli- 
cies—can cause friction in local communities. For example, as one community college ad- 


ministrator explained: 


One of the big differences between the community college and the school 
district is we don’t get very much money. We get a fairly small allocation, 
and there is really not enough money to actually do what is required. So we 


leverage funds all over campus and try to make this work. The Perkins mon- 
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ey is so critical to our campus. It is used to support extra lab hours, instruc- 
tional techs in the labs, and equipment purchases. Without the Perkins dol- 


lars, we could not have some of our programs. 


Allocating Funds to LEAs 
All six states use an application process for channeling funds to LEAs (Exhibit C.2). The 
applications generally parallel the states’ applications to the Department, usually cover five 
years, and often include a performance improvement plan. As a CTE specialist in one state 
observed, “The nice part with Perkins IV is that the locals are now mirroring what we do at 
the state level, which includes a plan revision each year which lets us address what changes 


we're making with budget and projections.” 


Exhibit C.2. 
State allocation process for LEAs 


State Allocation process for LEAs 
A e ~—- Allocations to LEAs and consortia based on formula 


e Each eligible recipient must submit an application 


B e Online application mirrors the application submitted by the state 


e = Application components include five-year plan, end-of-year report, performance improve- 
ment plan, and explanation of how LEA will address required and permissive uses of funds 


e Districts can apply individually or as part of consortium 


Cc e Web-based application that parallels the state plan 


e Plan can be changed each year 


D e = Online application process 
e = Allocation by formula 
e —_ Recipients must show they are meeting all requirements 
Ee Funding allocated to 15 secondary and technical education regions by formula. 
e Five-year plan must be submitted 


F ¢ = Online application process 


e Plan must demonstrate program quality and POS linked with postsecondary education 


Overall, local communities were satisfied with the process used by their states to allocate 
funds. All of them said, however, that funds for CTE were insufficient in their districts. As 
one CTE director noted, “The process we use is to discuss funding needs in the spring. We 
have big needs, and our Perkins allotment will not meet these needs.” Another local CTE di- 
rector stated, “We don’t have enough money to do everything. We could put all of it at one 
school and help them. It’s like anythine—we just have to kind of share the wealth.” 


Most states provided CTE funding to local districts in addition to Perkins funds. Not surpris- 


ingly, local districts in these states were less concerned about the level of Perkins funding and 
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tended to use the Perkins funds for equipment or special projects. In the one state that did 
not provide categorical funding for CTE, Perkins funding allocations to local districts were 
reported to be very important. As state staff emphasized, “We could not survive without 
Perkins. Our program survived because of the small amount of money that we have from 


Perkins, and we do the best we can.” 


Many LEAs had their own distinct processes for allocating funds. Some districts divided 


funds evenly among schools, while others focused on different CTE programs each year. 


Local school districts used the funds they received in different ways. One large school dis- 
trict provided significant local funding for CTE. “Perkins may come across to some districts 
as an unfunded mandate,” the district CTE director said. “We get a little over $3 million in 
Perkins fands, and I am running a $52 million dollar program.” In contrast, a medium-sized 
district in the same state provided little additional funding, and as a result, Perkins funds were 
used to cover administrators’ salaries and equipment for programs. In a smaller district in 
that state, Perkins funds were used for salaries and equipment. The CTE director in this dis- 
trict noted that his system has “dropped programs over the years because we don’t have the 
money.” Nonetheless, state and local funds represented the vast majority of CTE budgets in 


all three communities because they covered costs for teachers and facilities. 


Many LEAs reported forming consortia, allowed under Perkins I and continuing under Per- 
kins IV, in an effort to make greater use of limited funds. The minimum award to an LEA 
allowed under Perkins IV is $15,000. LEAs are encouraged to form consortia to operate pro- 
grams of sufficient size, scope, and quality to be effective. Funds allocated to consortia must 
be used for purposes and programs that are beneficial for all consortium members and can- 
not be re-allocated to individual members. “I think we have 12 schools in [our] consortium,” 
one local administrator (local A3) said. “It fluctuates. This particular year, the agriculture 
program has had the bulk of the funding, but we have used the consortium for counselor 


training, staff development, and assessment materials.” 


Allocating Funds to IHEs 
As with secondary allocations, all states used an application process to allocate postsecond- 
ary Perkins fands (Exhibit C.3). The applications mirrored states’ applications to the De- 
partment and covered five years. Often the annual application updates included quality 


assessments ot performance improvement plans. 
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Exhibit C.3. 
State allocation process for IHEs 


State Postsecondary allocation process 
A e = Allocation to seven IHEs by formula 


e Each eligible recipient must submit an application 


B e = Online application mirrors the application submitted by the state 


e = Application components include five-year plan, end-of-year report, performance improve- 
ment plan, and explanation of how LEA will address required and permissive uses of funds 


e = JHEs can apply individually or as part of consortium 


eb-based application that parallels the state plan 
e = Allocation by formula 


e Plan can be changed each year 


D e Online application process 


e —_ Recipients must show they are meeting all requirements 


E e = Funding allocated by formula to two IHEs 


e Five-year plan must be submitted 


F e = Online application process 


e Plan must demonstrate program quality and POS linked with secondary institution 


Some states used separate review teams to evaluate postsecondary and secondary applica- 
tions. One state, for example, convened a review team composed of Tech Prep and work- 
force education administrators and college deans to review applications and make 
recommendations. “The review process is great professional development,” the state Perkins 
IV administrator said. “We try to make sure there is a mix of new as well as experienced 
people, and they all learn a lot. Those [reviewers] that are new, definitely their plans are 


much better the next year.” 


States differed in the number of IHEs they fund, from two to more than 100. In one state, 
there was concern about the postsecondary formula because it uses the number of students, 
rather than FTE students, as the basis for allocation decisions. This process results in a 
smaller allocation to the IHE with more CTE programs. State officials requested a waiver, 


but it was denied by the Department. 


Many local administrators said they felt far removed from the allocation process in their 
states. Others, however, said that they participated and could control the allocation of funds 
locally. One Tech Prep ditector observed: “All of us have to develop a plan for how we’re 
going to divide funds among local programs, and so it’s actually pretty easy for us at the 
technical college, because we’re all professional technical education. We do a program effec- 
tiveness process. We all develop our Perkins plans, and we get the check, [which is then allo- 


cated among programs].” 
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In the same state, another Tech Prep consortium followed a similar process. “They deter- 
mine ahead of time funding allotments by consortia, dependent upon [the] Tech Prep head 
count from the previous year,” one community college Tech Prep director said. “So we have 
an idea about how much money we should be getting. Then I’m responsible for working 
with my board, and writing the grant, and addressing each of the four or five different areas, 


including the new goals for this year, and how are we going to meet them.” 


Allocating Funds to Area CTE Centers 
Three states allocated Perkins funds to area CTE centers (Exhibit C.4). One state allocated 
postsecondary Perkins funds to these centers. Another used state funds to support secondary 
programs, and additional postsecondary Perkins funds went to the centers to support adult 
students in the area CTE centers. The third state allocated all secondary funds to area CTE 
centers, although the state required each center to redirect a portion of its allocation to an 


IHE for CTE teacher preparation. 


Exhibit C.4. 
Allocating funds to area CTE centers 


State Allocations to area CTE centers 


C e Postsecondary funds are split between community colleges and adult schools or atea CTE 
centers with adult students based on enrollment counts from the past two years. 


e Area CTE centers do not receive secondary Perkins funding from the state. 


e A new stipulation in the state plan allows LEAs to form a consortium and release their Per- 
kins funds to an area CTE center to support CTE programs not offered in the districts. 


D e Area CTE centers receive both secondary and postsecondary Perkins funds—secondary for 
high school students and postsecondary for adult students who take courses in the centers. 


E e = All secondary Perkins funding goes to area CTE centers. 


e Area CTE centers also receive categorical funding from the state based on FTE as well as 
tuition from sending districts. 


e Perkins funds constitute 5 percent or less of each center’s annual budget. 


NOTE: Three of the six case study states had area CTE centers. 


Area CTE center directors and CTE directors are creative in their use of Perkins funds. They 
often combine them with other funding sources, or they use Perkins funds to supplement 


specific program or staff costs. As one area CTE center director reported, 


The tech center recetves both secondary and postsecondary funding. Perkins 
funding for adult education (Title I) comes directly to the tech center, while 
the secondary is shared with the high school. Annually, the tech center re- 
ceives twice as much for adult students as secondary students. We have 
both high school and adult students going to Career and Technical Student 
Organization (CTSO) competitions—40 percent adult, 60 percent second- 
ary students—and we use Perkins funding to support these students. The 
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other thing I look for, before I spend Perkins funds, is if I am receiving dis- 
trict, capital outlay, or other grant money that can be used to support the 


program. Then I don’t need to spend the Perkins funds on it. 


Funds for POS Dual Enrollment 
Even though Perkins IV placed considerable emphasis on POS as a new strategy for con- 
necting secondary and postsecondary education, the law did not explicitly direct funds for 
this purpose. As part of the definition in Sec. 122, Perkins IV states that a POS “may include 
the opportunity for secondary students to participate in dual or concurrent enrollment pro- 


grams or other ways to acquire postsecondary education credits.” 


Considerable variation was found across the six states in their financial support for dual en- 
rollment programs and articulation agreements. Three states did not provide any funding for 
dual enrollment, and representatives from these states reported that dual enrollment was 
problematic because it was difficult to determine how funds would be allocated between 
secondary and postsecondary institutions. In one state, the fear of “double dipping” was cit- 


ed as a barrier to POS implementation in some sites. 


Three states, however, did provide funding for dual enrollment. One state offered LEAs and 
IHEs three alternative strategies. In the second state, all costs were covered by the area CTE 
centers, and in the third state, lottery funds were used to reimburse IHEs, while LEAs tre- 


ceived the student allocation (Exhibit C.5). 
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Exhibit C.5. 
State strategies for dual enrollment funding 


State Dual enrollment funding strategy 
A e No dual enrollment funding 


D e = Three different funding strategies are used: 
1. Both LEA and IHE absorb costs independently. 


2. IHE reimburses LEA for instruction delivered on high school campus by 
high school teachers. 


3. LEA reimburses IHE for instruction delivered by community college instruc- 
tors on high school campus. 


E e Area CTE centers cover all costs and receive the student allocation. 


e = Dual enrollment courses are approved by the IHE and taught at the area CTE 
center by instructors paid by the center. A student fee often is covered by the area 
CTE center as well. 


F e — Students counted as part of LEA average daily attendance at secondary level. 


e —_IHEs reimbursed with funds from a state lottery. 


Minimum Allocation 


Perkins IV permits states to waive the minimum allocation rule if an LEA is located in a ru- 
ral, sparsely populated area; if it is a public charter school that operates a secondary CTE 
program; and if it demonstrates that it is unable to join a consortium. Similarly, Perkins IV al- 
lows state administrators to grant waivers to [HEs if they are located in rural, sparsely popu- 


lated areas. 


States used waivers to address different circumstances. Officials in two states reported using 
waivers for LEAs that were unable to meet the minimum allocation requirement. In one 
state, Perkins administrators said that they had granted waivers to a few districts based on 
their rural locations. A secondary CTE administrator in this state reported that about 40 
LEAs were routinely granted waivers until state officials changed the definition of rural and 
the number of waivers decreased. He noted that Perkins administrators subsequently asked 
the state board of education for further revision of the rural definition to enable more LEAs 
to qualify for waivers and described an electronic waiver request form that requires local 


administrators to document efforts to form or join a consortium. 


A local CTE director in a small, rural district in this state offered firsthand experiences of the 
challenges associated with the changes in the state’s approach to waivers. He explained that 
his district previously was part of a consortium for which the area CTE center served as the 


fiduciary agent. “When Perkins IV came around, the [center] decided to disband the consor- 
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tium, and districts were told they could apply for waivers,” he recalled. His LEA initially did 
not qualify for a waiver based on rural location because state data included it in a standard 
metropolitan statistical area. He said that he “made a case for a waiver,” citing three factors: 
most schools in the region qualified for the minimum allocation; they were too far away for 
effective collaboration; or they did not want to write a CTE plan. Even though state officials 
did not accept his contention that the LEA was rural, the district recetved a waiver based on 
its inability to form a consortium. He also noted, “A year later, [the department of educa- 
tion] made an announcement reopening Perkins doors to other small districts to seek waiv- 
ers.” 

Two states reported granting waivers to IHEs that were unable to qualify for the minimum 
allocation. A high-ranking postsecondary administrator in one state suggested that some 
IHEs use consortia strictly as funding entities and an alternative to obtaining formal waivers: 
“Every year, we have a couple of colleges that are right on the edge of qualifying for the Per- 
kins funding. Because they are in geographically isolated rural areas, we’re able to invoke the 
waiver that allows them to run their own programs separately. It’s just that they are in a con- 
sortium as a funding entity; it is not a true consortium, because they’re not serving the same 


populations.” 


While this arrangement seems to run counter to Perkins language limiting spending by con- 
sortia to purposes that benefit all members, administrators in the other state reported what 
seemed to be stricter adherence to waiver requirements. According to one state official, 
postsecondary institutions that are unable to qualify for the minimum allocation “would 
have to enter a consortium to receive those dollars. They would be given an application 


which details the requirements and provides the waiver information.” 


Reserve Funds 


Perkins IV has a reserve option allowing states to distribute up to 10 percent of the state al- 
lotment that goes to the local level by means other than the formulas detailed in Sec. 131 and 
132. These funds may be made available for CTE programs in areas that are rural and have 


high percentages or high numbers of CTE students. 


Four states took advantage of the reserve option. The primary reasons offered by the states 
were to increase flexibility, make additional funding available to rural communities, and sup- 
port innovative programs and services through either competitive grants or use of formulas 
(Exhibit C.6). As one administrator stated, “I like the reserve option because it gives systems 
that don’t normally receive a large sum of money the opportunity to put a program into 
place. It also gives encouragement for systems to be innovative and look at new labor market 
data and not just continue to offer the same CTE programs that they’ve offered for years.” 


Some states appear to have exercised some discretion in distributing grants under the reserve 
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option. State officials in one state reported using reserve funds to support a number of pro- 


grams that did not seem to fall under the three categories specified in Perkins IV. 


Exhibit C.6. 


State uses of reserve funds 


State Use of reserve funds 
A e Funds distributed through competitive innovation grants and formula-based programs. 


e Funding supports innovation, development of resources, and best practice models. 


B ° Funds go to rural districts with large CTE enrollments. 


e Approximately 150 districts received support through reserve funds. Reserve funds are dis- 
tributed to colleges based on a prorated share allocated to colleges in counties with popula- 
tions of less than 100 people per square mile. 


e Reserve funds go to two groups: (1) university lab schools, state school for the deaf and 
blind, and department of juvenile justice districts and (2) rural and sparsely populated areas, 
by formula. 


e The latter group of LEAs must work with colleges and focus on one of four areas: distance 
learning; IT; priorities set by the workforce board; or support for teacher or student certifica- 
tions. 


F e — Allocation is distributed 80 percent to secondary and 20 percent postsecondary education. 
e Competitive grants used to support secondary to postsecondary transition programs for 
POS development, articulation, dual credit, dual enrollment, and distance learning. 


e Funds support approximately 24 grants per year, ranging from $5,000 to $100,000 with an 
average of about $70,000. 


NOTE: Four of the six case study states made use of the reserve option in Perkins IV. 


While not all state CTE administrators decided to use the reserve option, those who did cit- 
ed increased flexibility and support for innovation in CTE as important benefits. A state 
administrator in a state not currently using the reserve option described possibly using its 
merged Tech Prep funds to establish a reserve fund for competitive grants for POS devel- 
opment. The CTE director in the other state not using the reserve option explained, “Our 
state hasn’t been willing to get into having a reserve fund, because we don’t want to be per- 


ceived as taking money away from districts.” 


State Leadership Funds 


States can devote up to 10 percent of their Perkins funds to state leadership activities. States 


are required to apply these funds to nine different purposes and 17 permissible uses. 


Further, states must use leadership funds to serve individuals in state institutions and prepare 
students for nontraditional fields. Perkins IV also allows states to reserve up to 5 percent of 
their total state allotment or $250,000—whichever is greater—tfor administration of the state 
plan. When using money for these administrative purposes, states are required to provide 


dollar-for-dollar matching funds. 
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Officials in all six states said they used leadership funds for a range of programs and services 
(Exhibit C.7). Some state directors were not asked about all nine required uses and men- 


tioned only a few of the permissible uses, so it should not be assumed that these are the 


states’ only uses of the funds within the two categories. 


Exhibit C.7. 
Required and permissible uses of leadership funds 


Required Permissible 

e Professional development (All six e — Statewide articulation agreements 
states) (States A and D) 

e Technical assistance (All six states) e = Support for CTE student organizations 


e = Support programs for special popula- pie kG end) 
tions (States A and D) e Incentive grants (State C) 


e Preparation for nontraditional fields e Improve academic counseling (State D) 


(All six states) e Develop new CTE courses (State D) 


e Assessment of CTE programs (State E) © — Support for POS development (States 


e Contracts for assistance in developing D and F) 
ee es, aes ¢ Co-sponsorship of special events (State 
design, piloting, and statistical review, E) 
as well as co-facilitation of meetings 
with faculty) (State E) e Website focusing on nontraditional 
students and career guidance programs 
(State F) 


One state provided state leadership block grants to IHEs to support required and permissi- 
ble uses, including replication of best practices. IHEs in this state could also apply through a 
competitive process to receive funding to support new and innovative best practices. Anoth- 
er state’s leadership funds were primarily distributed to support statewide or regional pro- 
jects, including content advisory committees that bring together industry representatives and 
faculty to discuss trends in a specific discipline and determine priorities for curriculum and 
professional development. Other leadership funds in this state were distributed through 
competitive grants to community colleges, with the intent of providing an incentive to part- 


ner. These funds went to regional consortia to foster program coordination and information 


sharing. 


Administrators in several states indicated that the amount allowed for leadership funds was 
inadequate to meet required and permissible uses. One state official mentioned several criti- 


cal activities, including teacher preparation, on which they would like to spend leadership 


dollars—if they had more funding. An administrator from another state noted: 


I think there has to be recognition that, with the requirements for this par- 
ticular legislation . . . there has to be sufficient administrative funds [or] the 


leadership portion ought to be increased. Strictly from a state perspective, 
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you need sufficient leadership and administrative dollars to make this pro- 
gram do all that it could do and should do. If Perkins IV had some oppot- 
tunity for increased amounts that could be retained at the state level, we’d 
be in favor of that, especially since in our state these dollars are more dis- 


cretionary and flexible. 


Nontraditional Training and Employment 


Administrators in all six states said that they spent more than the required $60,000 on non- 
traditional training and employment. Respondents offered information on both state-level 
suballocations for nontraditional training and employment and specific examples of local us- 


es of these funds (Exhibit C.8). 
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State 


Exhibit C.8. 


Nontraditional training and employment suballocations 


Expenditures 


State allocated $60,000 and reported investing more than that on nontraditional 
training and employment activities. 


A local CTE director developed a series of “nontraditional posters” with photos 
and examples of people in nontraditional careers. 


The state appropriated $150,000 for nontraditional training and employment activities. 


Funds were divided across the state’s colleges for a few targeted activities to inform students 
of nontraditional careers. 


A local CTE director did not set aside funds for nontraditional training and employment 
but exposed students to an array of careers through posters, software, career days, job fairs, 
and job shadowing. 


Local LEA staff helped coordinate events to promote nontraditional careers. 


State allocated $150,000 for nontraditional training and employment activities. 


State provided resources, outreach, and training to educational practitioners at both K—adult 
and community college levels. The nature and amounts of expenditures did not change dra- 
matically since Perkins LII. 


One LEA hosted annual nontraditional career fairs because it did not meet the non- 
traditional performance targets. 


Another LEA spent funds on recruitment materials and activities for outreach to nontradi- 
tional populations and developed business partnerships to secure internships and mentor- 
ships for students. 


State allocated $60,000 for nontraditional training and employment activities. 


The state funded 40 percent of a state staff person’s time for nontraditional training and 
employment and special populations. 


A local CTE director developed promotional posters, a kit to distribute to schools, and ex- 
loratory summer camps to promote nontraditional careers 


tate ocate: $ or nontraditiona training and emp. oyment activities. 


The state CTE director deliberately sought business associations to help with 


outreach to students. 


A local CTE director was planning a video recruiting campaign aimed at nontradi- 
tional students. 


State allocated $150,000 for nontraditional training and employment activities. 


The state created an annual program that awards a $500 reimbursable scholarship to students 
in nontraditional programs. 


State and local CTE directors expressed concerns about the nontraditional training and em- 


ployment requirement and offered recommendations to improve it (Exhibit C.9). State offi- 


cials from two states suggested examining the current status of nontraditional training and 


employment to ensure more complete understanding of the barriers faced by students enter- 


ing certain fields. They also asked that federal officials provide them with more effective 


strategies and best practices at the federal level. One local CTE director suggested deleting 


this requirement from Perkins IV altogether: “I would get rid of nontraditional in a heartbeat. 
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Does it matter whether they’re concentrating in a nontraditional area or gaining the skills to 
succeed in any area? I would like to hear what kind of difference that makes, if you’re trying 


to make sure that, at the end of the day, every student has mastered the competencies.” 


Two local CTE directors said that although they can get students to take nontraditional 
coutses, it is challenging for them to complete the program of study because as they “they 
see fewer and fewer students of their same gender as they continue through the POS, and 


they become discouraged and drop out of the program.” 


Exhibit C.9. 
Perceptions of nontraditional training and employment requirement 


State Perceptions 


A e = State CTE director suggested doing an in-depth evaluation of progress on nontra- 
ditional training. The director did not feel that guidance from the Department was 
very substantive and would like to have the Department as a resource on what 
works. 


e = A local CTE director said some students who take nontraditional courses do not 
continue through the POS. Further, it is critical to connect with students much 
earlier than high school to encourage them to think outside of gender stereotypes. 


B e A local CTE director noticed he no longer has to recruit students actively to take nontradi- 
tional courses; he has seen progress, with more students taking nontraditional courses. 


e State CTE director expressed need to start these initiatives earlier (in elementary 
school) and involve parents in promoting nontraditional participation. He agreed 
that it is important to look at the data on nontraditional training and employment 
but suggested the criteria should not be so high-stakes and LEAs and IHEs 
should not be sanctioned. In addition, he mentioned barriers for students, includ- 
ing harassment in classrooms and on the job and family pressures against pursuing 
a nontraditional career. 


D e = State CTE director felt that the nontraditional training and employment require- 
ment is outdated and noted that the state exceeds the national average in terms of 
students taking nontraditional courses in certain areas. 


E e =A local CTE director reported exceeding state goals for gender equity and said 
that that the sooner you can reach the students the better, which is why they have 
programs in the middle school. 


F e =A local CTE director identified challenges in retaining nontraditional students because of the 
stigma attached to coutses typically taken by students of one gender—and mentioned it 
would be helpful to look at current labor market data to see what progress is being made. 


State Administrative Funds 


Perkins IV allows states to allocate funds for state administrative purposes, including devel- 
opment of state plans, program monitoring and evaluation, technical assistance, and devel- 
opment of data systems. Administrators in five states reported spending these funds on staff 
salaries. In three states, funds were spent on staff travel for program monitoring and tech- 


nical assistance. One state used administrative funds to gather student data, and another used 
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funds for salaries for data staff. In other states, administrators reported using funds to de- 


velop, review, and implement state plans. 


Perkins IV stipulates that all states must match, from nonfederal sources and on a dollat-for- 
dollar basis, funds spent on administrative activities. State CTE directors reported that meet- 
ing the matching requirement was a challenge and shared their strategies for identifying the 
necessary funds. One state administrator described using salaries for corrections staff as part 
of its match, because those staff are paid with state dollars to administer CTE within the 
prison system. Administrators from two states reported that the match requirement is met 
by the state general fund or general revenue spent on administrative functions by state agen- 
cies. Administrators from three states expressed concern about meeting the match require- 
ment. One state CTE director mentioned that the match requirement limits the state’s ability 


to use state administrative funds and acknowledged it has been a struggle to meet the match. 


Across the states, the amount of funding for administrative uses (i.e., Perkins IV funds plus 
matching funds) was widely viewed as inadequate to meet administrative needs. A state CTE 
director reported that the amount allowed for administrative costs had remained constant, 
while actual costs had continued to increase. As a result, fewer funds were available for in- 
novation and leadership functions. 


Some administrators suggested that the additional requirements in Perkins IV should have 
triggered additional administrative funding to help with implementation. One state CTE di- 
rector explicitly stated that administrative dollars were insufficient to meet the federal re- 
quirements. Another CTE director reported, “We struggle. We’ve been able to meet it to 
date, but cuts at the state level reduce your general revenue dollars. So one day we’re going 
to be at a point very soon that you won’t have that same level of match... . [W]e struggle 
with the documentation of it. We’re constantly doing this staff analysis to make sure that 


we’te able to maintain our match.” 


Recommendations for Future Changes to Funding Provisions 


State and local CTE administrators, faculty, and staff offered numerous recommendations 
for changes to the funding provisions in future Perkins reauthorizations. Common themes 
included a need for more funds, a desire for greater flexibility in allocating funds, and an in- 
terest in modifications to the formula for distributing the Perkins basic grant (Exhibits C.10 
and C.11). 


Most state administrators advocated for increased Perkins IV funding and greater flexibility, 
with a number noting that they are looking increasingly to the federal government for CTE 
funding as state and local budgets continue to be squeezed. Several noted that the increased 
accountability and leadership requirements were not accompanied by parallel increases in 


funding. Some recommended additional funding for teacher professional development and 
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training (particularly industry certifications), new software training, and conference travel to 
connect with industry representatives and employers. Several also said that more Perkins 
funds were needed to cover rising costs of equipment related to new high-tech, high-skill ca- 
reers. One secondary state CTE director said, “P’ve reached the conclusion now that we 
probably need twice as much Perkins money. There’s a huge need at the K—12 level and a 
huge need at the community college level, and when you split it up, it gets watered down 


very quickly.” 


Most state staff indicated the federal formulas can be constraining and suggested that greater 
flexibility would be beneficial in the long run, both in terms of those served and in program 
outcomes (Exhibit 10). Some state administrators recommended reducing the administrative 
and reporting burden attached to funds or developing different requirements for large and 
small districts because they receive different amounts of money but are still held accountable 
to the same standards. Another common recommendation related to the administrative 
match. Many suggested that the requirement that state funds come from general revenue is 


too constraining (Exhibit C.10). 


Finally, several state administrators indicated that the Perkins allocation formulas should be 
modified to reward high levels of CTE participation. They noted that school districts and 
postsecondary institutions with large numbers of CTE students are negatively affected if 


most of these students are not living in poverty. 
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Exhibit C.10. 
State recommendations for changes to Perkins funding provisions 


State Recommendations Quotes 
A e Link CTE to state economic development e §=Our ability to help economic development is ham- 
: i me m : 
© Increase CTE funding pered every time we lose some money 
: ; : ° y t ropriat 
e ~—_ Explore alternative funding allocation formulas Tspeveny the aaa ealipest app ae 
funds for CTE? It is not a poverty program; it is an 
education program. 
B e Increase funding e Increase funding to keep pace with increasing ac- 
e Increase flexibility eovatbiinys 
inistrati 7 xibility 1 ide for uni 
é Shibhenes Giese tua Flexibility is necessary to better provide for unique 
programs. 
e Maintain Tech Prep as a separate funding provision : Se 
P P SP e =The mote discretion in how to use the dollars, the 
e Separate outcomes from programs better the results. 
e Increase support for small and medium communi- e There has to be sufficient administration, and if it is 
ties too crazy to increase administration to over 5 per- 
cent, the leadership funds should be increased. 

e Give us the outcomes, but let us take the money to 
move up to where we need it to accomplish the out- 
comes. 

e = Our small and medium districts face huge challenges 
because there is so much they have to do with so lit- 
tle support. 

Cc e Modify the formula to “incentivize” CTE e The current federal formula for funding allocations 
: Sani istricts i rt t t 
© Increase local funding flexibility to local districts is based on poverty counts, no 
CTE enrollment. 
e Increase length of time for waivers tee . 
= e There ought to be some flexibility in strategically 
e Change maintenance of effort requirement investing funds that go to LEAs based on some 
e = Lower the postsecondary minimum allocation to state formula. 
$40,000 e We haven’t used a waiver because waivers ate for 
e Ensure that secondary and postsecondary voices are only one year. 
heard e I would change the maintenance of effort to match 
what’s in adult education, IDEA, or ESEA so you 
can fluctuate within 10 percent in any year. 

e@ = Small rural colleges wouldn’t have to worry about 
the minimum, and they wouldn’t lose 5 percent that 
goes to the consortium. 

e The postsecondary side has zero status—the De- 
partment only contacts the secondary CTE director. 

D e Increase federal share of CTE funding e —_There’s never enough. I think over time the needs 
e Provide greater flexibility for the administrative a ea 
match e We have general revenue, but the way the law is 


written you can only use general revenue that is 
spent on administrative functions and by state agen- 
cies to do your dollar for dollar match. ... We’ve 
been able to meet it to date, but cuts at the state lev- 
el reduce your general revenue dollars. So one day 
we're going to be at a point very soon that you 
won't have that same level of match. 
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Exhibit C.10.—cont. 
State recommendations for changes to Perkins funding provisions 


State Recommendations Quotes 


E ° Increase the small state minimum e Our state gets a small state minimum, and it has not 
gone up since Perkins II. 


° Fix the funding and governance structures 
e Fixing the funding and governance structures might 
e Change postsecondary funding formula lead to higher CTE enrollments. 


e The postsecondary formula is problematic because 
it is based on number of students rather than FTEs. 
F e Agree on national priorities for CTE e [Identify] what we are trying to accomplish and 
focus funding specifically on those things. 
e Focus on outcomes and data systems 
e The formula needs to be reexamined in light of 
e Increase federal funding what we need to accomplish. 


e Increase support for equipment e ~=We have added to the legislation, but we haven’t 


; ; added the dollars to support it. 
e = Link CTE funding to labor market data 


e When the equipment gets old and you don’t have 
money to suppott it, the next thing that happens is 
the program is closed. 


e = Maintain the increased flexibility of reserve grants 


e Provide incentives and strategies to attract CTE 


teachers e [There] should be so much more emphasis on CTE 


: than what exists today. 
e Increase focus of CTE to meet needs of diverse y 


groups of students e The flexibility allowed our postsecondary institu- 


tions to be more innovative. 


e = Many people will not give up a high-paying job for a 
$30,000 CTE teaching position. 


e — If you cut out Perkins, you’re going to see a higher 
dropout rate. 


Common recommendations from local CTE administrators and faculty included providing 
additional funds for professional development and equipment and reducing administrative 
burden. Some local representatives specifically mentioned providing additional funds to pur- 
chase, install, and maintain equipment to ensure that students are using the latest technology. 
Other local recommendations included reducing funding delays to bring the actual receipt of 
funds in closer alignment with expenditures and increasing funds for innovation in pro- 
gramming (Exhibit C.11). 
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Exhibit C.11. 
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Local recommendations for changes to Perkins funding provisions 


State Recommendations Quotes 
A e Provide funds for equipment installation and e [We] need to expose youth to CTE opportunities 
maintenance earlier. Unlike smaller districts where K-12 students 
* “Allow ude af Bands a0 expose poutiger youth to are in the same building, larger districts have fewer 
CTE opportunities for younger students to be exposed to 
CTE. 
Allow fundi iddle schools for student F 
: = Neti touniddle: schools for sedenicateet e Funds should be spent to train counselors on Per- 
P kins IV, as some have retired and new counselors 
e Increase funding for professional development and do not have background or training in Perkins IV. 
training for CTE instruct : 
Cg eects e There is a lack of funds to respond to the technolo- 
e Provide funds to update equipment gy explosion in many CTE courses, which necessi- 
e Include business and industry representatives in tates the reliance on technology at the community 
funding allocation processes college. 
cau ee de me eal Se ee 
development enhance the ability of teachers. 
e = Direct funds to curriculum development and inno- e = It would be helpful to have more flexibility to use 
vation the funds to pilot a new curriculum or hire high- 
© Increase flexibility demand instructors. 
eo © Regionalize Perkins IV funds to eliminate need for © Districts should consider declaring intent to use the 
multiple local CTE plans funds, so the state knows which districts are willing 
e ~~ Reduce administrative and reporting burden at- . os tithe viene fer clevelap a plan and eati the 
tached to funds ane 
: ; ? Funding level is usually based upon average dail 
; hool * Ss y Pp 8 y 
pe Veeyee du ireiniears depen dipaieneshoohaue attendance (ADA) or school size, but the require- 
ments ate the same whether it is a small, rural 
school or a large, urban district. 
> See aaa eee aaa Se ee 
and rural districts velopment because technology is changing rapidly 
© — Make seed money available through a competitive and instructors need to be able to teach at that level. 
process to fund innovative programs e This small, rural area receives significantly fewer 
e Increase funding for CTE instructor professional ee ae ee ae 
ble to the same standards. 
development 
e Provide more funds for teachers to attend confer- 
ences for professional development and to link with 
industry employers 
e Supply funds for industry certifications and soft- 
wate training for CTE instructors 
e §=©Minimize funding distribution delays 
e Increase flexibility 
e 


Develop different accountability standards for large 
districts and small, rural districts 
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Exhibit C.11.—cont. 
Local recommendations for changes to Perkins funding provisions 


State Recommendations Quotes 
E e — Allow larger area CTE centers to benefit more di- e There is competition between secondary and post- 
rectly secondary, and technology centers do not receive 


° Increase funding level adequate funds. 


F e Provide funds to update equipment, specifically e We want to expand the program so that every stu- 
computer technology dent [who] wants to take a CTE course has the op- 


° Do not allocate funds through RFP process portunity: 


e There is a discrepancy between the date when the 
county is allowed to begin spending funds, and 
when the funds are actually received, presumably 

e Minimize funding distribution delays due to the difference between local and federal fis- 

cal years. 


e — Incentivize cooperation between secondary and 
postsecondary, and even within postsecondary 


Implementing Perkins IV Accountability Provisions 


Perkins IV maintained the central accountability components introduced in Perkins LI, 
heightened emphasis on coordination with other federal statutes and programs, and added 
greater specificity and higher stakes with regard to core indicators, reporting requirements, 
and negotiated performance levels. This section provides a summary of the major accounta- 
bility provisions in Perkins IV and describes the status of their implementation in case study 
states and local communities. Particular attention is paid to the variation across and within 


states and on the differences found between the secondary and postsecondary levels. 


Major Accountability Provisions in Perkins IV 


While Perkins III listed only four core indicators for use at both the secondary and postsec- 
ondary levels, Perkins IV identifies different indicators for each level. Under Perkins IV, eligi- 
ble agencies are required to identify performance measures for each core indicator. Moving 
well beyond the Perkins III statutory language that allowed use of “currently identified state 
performance measures that meet the purpose of the Act” (ACTE 2006, p. 104), Perkins IV 
explicitly states that “an eligible agency shall, to the greatest extent possible, align the indica- 
tors so that substantially similar information gathered for other federal and state programs. . 
.is used to meet the requirements of this section” [Sec. 113(b)(2)(P)]. 


At the secondary level, Perkins IV mandates use of the high school academic assessments te- 
quired under ESEA and requires reporting of graduation rates as described in ESEA. Statu- 


tory core indicators of secondary student performance are: 
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e Student attainment of challenging academic content standards and student aca- 
demic achievement standards, as adopted by a state in accordance with. . . 
ESEA and measured by the state-determined proficient levels on the academic 
assessments described in. . [ESA]; 


e Student attainment of career and technical skill proficiencies, including student 
achievement on technical assessments, that are aligned with industry-recognized 


standards, if available and appropriate; 


e Student rates of attainment of each of the following: a secondary school diplo- 
ma; a General Education Development (GED) credential or other state- 
recognized equivalent (including recognized alternative standards for individuals 
with disabilities); [and] a proficiency credential, certificate, or degree, in conjunc- 
tion with a secondary diploma (if such a credential, certificate, or degree is of- 


fered by the state in conjunction with a secondary school diploma); 
e Student graduation rates (as described in. . .A.S'EA); 


e Student placement in postsecondary education or advanced training, in military 


service, or in employment; [and] 


e Student participation in and completion of CTE programs that lead to nontradi- 
tional fields. 


At the postsecondary level, Perkins IV eliminates the Perkins II academic attainment indica- 
tor and splits the Perkins II placement and retention indicator into two indicators, one 
measuring retention in postsecondary education and the other measuring placement in em- 
ployment, the military, or apprenticeships. The statutory indicators of performance for post- 


secondary students are: 


e Student attainment of challenging career and technical skill proficiencies, includ- 
ing student achievement on technical assessments that are aligned with industry- 


recognized standards, if available and appropriate; 


e Student attainment of an industry-recognized credential, a certificate, or a de- 
gree; 

e Student retention in postsecondary education or transfer to a baccalaureate de- 
gree program; 


e Student placement in military service or apprenticeship programs or placement 
or retention in employment, including placement in high-skill, high-wage, or 


high-demand occupations or professions; [and] 
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e Student participation in and completion of CTE programs in nontraditional 
fields. 


Perkins IV also raises the bar for state and local reporting on these core indicators and adds 
new emphasis on performance at the local level. Perkins III required states to negotiate quan- 
tifiable levels of performance for each core indicator with the Department but stopped short 
of requiring detailed reporting on each indicator. Perkins IV mandates disaggregated report- 
ing on each indicator for each category of students identified in ESEA as well as those iden- 
tified as special populations in Perkins. In addition, the legislation requires eligible 
recipients—i.e., LEAs, area CTE centers, [HEs, or consortia at either the secondary or post- 
secondary level—to accept the state-adjusted levels of performance or negotiate their own 


unique levels of performance with the state’s eligible agency. 


Perkins IV specifies sanctions for failure to meet negotiated state or local performance levels 
and establishes a timeline for program improvement. A state or eligible recipient that fails to 
meet at least 90 percent of an agreed-upon target has to develop and implement a program 
improvement plan. Perkins IV also establishes provisions for withholding funds if the im- 
provement plan is not implemented, no improvement is shown, or the state or eligible recip- 


ient fails to meet at least 90 percent of a performance level for three years in a row. 


In 2007, the Department issued nonregulatory guidance to help states develop performance 
measures that meet Perkins IV’ requirements. This nonregulatory guidance outlines meas- 


urement population definitions as well as performance measures. 


The nonregulatory guidance was designed to help states “build a stronger and more valid 
and reliable accountability system for career and technical education across the nation” 
(Justesen 2007a, p. 1). States were given the option to propose other population definitions 
and measurement approaches in their Perkins plans, but those that opted to do so wete re- 


quired to describe how the alternative definitions and measures would be valid and reliable. 


Key Findings Related to Perkins IV Accountability Provisions 


Case study interviews at the state and local levels revealed strong support for the concept of 
Perkins accountability but questions about whether the core indicators are appropriate 
measures of performance. Nonetheless, most state and local CTE administrators and faculty 
appeared to be taking the new reporting requirements seriously and placing greater emphasis 
on collecting and reporting valid and reliable data. The sophistication of the Perkins account- 
ability systems varied significantly across (and sometimes within) the case study states, but all 
could be described as works in progress. Although state CTE administrators generally exhib- 
ited a thorough understanding of Perkins reporting requirements and a commitment to help- 
ing LEAs and IHEs provide valid and reliable data, local case study visits suggested 


significant differences in understanding of the state’s performance measures and the popula- 
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tions to be reported. Key findings related to implementation of the Perkins IV accountability 


provisions ate summarized below. 


e State CTE administrators generally appreciated the nonregulatory guidance pro- 
vided by the Department, but they liked the flexibility to create their own defini- 
tions and performance measures based on state data sources, reporting capacity, 


and program delivery structures. 


e All of the case study states used at least one of the Department’s recommenda- 
tions for measurement populations and performance measures, but none adopt- 


ed all of them verbatim. 


e Reporting capacity within the states ranged from sophisticated web-based por- 
tals to more rudimentary systems in which data were tabulated locally and sub- 


mitted via e-mail or CD. 


e Most state administrators expressed confidence in their ability to report dis- 
ageregated data on most of the special populations identified in Perkins IV, but 
local CTE practitioners were less sanguine about their capacity to provide in- 
formation about special populations, more explicit in their descriptions of the 
challenges associated with collecting them, and more candid about their reliance 


on soutces that could not be verified. 


e Many administrators at both the state and local levels said that the cost of Per- 
kins accountability systems far exceeds the human and financial resources avail- 


able for this purpose and thus poses a substantial burden. 


e State administrators often reported using local Perkins performance data to iden- 
tify programs in need of improvement, but local administrators offered varied 
opinions about the extent to which local performance targets actually led to im- 


proved performance. 


Defining Populations and Performance Measures 


Although Perkins IV emphasizes accountability through identification of a set of core indica- 
tors to be used by states and local subgrantees to assess program effectiveness, it does not 
stipulate how subgrantees should measure performance against these indicators. The De- 
partment required states to establish performance measures for each core indicator and in- 
clude these measures along with definitions of CTE participants and concentrators in their 
five-year Perkins plans, but the legislation provides no guidance on student definitions or 
measure construction. States were thus allowed to develop their own definitions and 
measures based on factors such as data availability, data collection capacity, or ptevious 


measurement approaches. The Department’s process for review and approval of Perkins 
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plans was designed to ensure that each state included valid and reliable measures for the core 
indicators of performance, but it could not ensure consistency of measurement approaches 


across States. 


Measurement Populations 
The Department’s nonregulatory guidance offers a set of definitions for CTE participants 
and concentrators at the secondary and postsecondary levels. Although neither of these 
terms appears in the Perkins IV legislation, CTE participants and concentrators are the two 
groups identified in the Department’s suggested performance measures. Typically, the defini- 
tions of these populations are based on the number of CTE courses completed or the num- 
ber of CTE credits earned. Variations in these definitions across states mean that states are 
reporting on different populations and therefore limit the comparability of reported perfor- 


mance levels. 


The Department recommended that a secondary student be identified as a “CTE pattici- 
pant” after earning one or more credits in any CTE program area and identified as a “CTE 
concentrator” after earning three or more credits in a single CTE program area (or two cred- 
its in program areas where two-credit sequences are recognized by the state or its eligible lo- 
cal recipients). The suggested postsecondary participant definition was virtually identical—a 
postsecondary/adult student who has earned one or more credits in any CTE program. As 
criteria for identifying postsecondary concentrators, the Department recommended either 
completion of at least 12 academic or CTE credits within a single program sequence or 
completion of a short-term CTE program sequence of less than 12 credits, with either se- 
quence terminating in an industry-recognized credential, certificate, or degree. Only one of 
the six states adopted the Department’s suggested population definitions at both the second- 
ary and postsecondary levels, and more states exercised flexibility with their postsecondary 
definitions than with their secondary ones. One state adopted two sets of postsecondary def- 


initions—one for degree-granting institutions and one for certificate or diploma programs. 


At least three of these states continued to use the term “completer” at one or both levels to 
identify a third population and distinguish between students who enroll in CTE courses at 
the level required for concentrator status and those who actually complete a CTE program 
ot sequence. In one of these states, the CTE director and a postsecondary administrator 
provided oral definitions of CTE completers, even though written definitions were not in- 
cluded in their state Perkins plan. Several states previously had included completers as fun- 
damental elements of their Perkins III accountability systems (White et al. 2004), and some 
continued to do so even though completers were not included among the populations de- 
fined in the nonregulatory guidance. A state’s decision to include completers instead of con- 
centrators in calculating performance poses an additional barrier to comparability of Perkins 


accountability data across states. 
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At the secondary level, differences between the definitions recommended by the Depart- 
ment and those adopted by case study states included emphasis on enrollment instead of 
completion of courses or use of course time instead of credits (Exhibit C.12). Two states 
used enrollment in CTE programs or courses to define participants. One of these states used 
completion of 350 hours of program instruction or half of the program’s required skill as- 
sessments to define a concentrator; the other used enrollment in two courses of courses 
above the exploratory level in a single career cluster and defined a completer as a student 
who completed a CTE instructional program. A third state used instructional time complet- 
ed to identify participants and enrollment in the second half of a course sequence or mullti- 
hour course to define concentrators. In this state, concentrators were also counted as partic- 
ipants. An administrator explained, “We’ve done that because we’ve created a data system 
that is based on the total number being the participants and everything else is a subset. When 


we look at our participants, those are all CTE students.” 


APPENDIX C. CROSS CASE REPORT C-36 


Exhibit C.12. 
Secondary student definitions 


Participant 

OVAE guidance A secondary student who has earned one (1) or more credits in any career and technical educa- 
tion (CTE) program area. 

State A A secondary student who has earned one (1) or more credits in any CTE program atea. 

State B A secondary student who has enrolled in one (1) or more courses in any CTE program area. 

State C A secondary CTE participant is a student who has completed the equivalent of a conventional 
50-minute class taken five times per week for 180 school days or approximately 150 hours of 
instruction in a state-recognized CTE sequence or program. 

State D A secondary student who has earned one (1) or more credits in any CTE program. 

State E A secondary student who is enrolled in a state-approved technical education program that 
addresses the core academic and technical competencies identified as needed for employment 
and/or further education in a career cluster. 

State F A secondary student who has earned one (1) or more credits in any CTE program area. 

Concenttator 

OVAE guidance A secondary student who has earned three (3) or more credits in a single CTE program area 
(e.g., health care or business services) or two (2) credits in a single CTE program area, but only 
in those program areas where 2-credit sequences at the secondary level are recognized by the 
state and/or its local eligible recipients. 

State A A secondary student who has earned three (3) or more credits in a single CTE program of 
study area (e.g., health sciences or business administration) or two (2) credits in a single CTE 
program area, but only in those program areas where 2-credit sequences at the secondary level 
ate recognized by the state and/or its local eligible recipients. 

State B A student who has enrolled in two or more CTE courses above the exploratory level in a 
single cluster. 

State C A secondary CTE concentrator is a student who has completed 50 percent of a planned pro- 
gram sequence (in hours or credits) in a state-recognized CTE sequence and is enrolled in the 
next course in that sequence or has completed 50 percent of a single, state-recognized, multi- 
hour coutse and is enrolled in the second half of that course. 

State D A secondary student who has earned three (3) or more credits in a single CTE program or 
two (2) credits in a single CTE program, but only in those programs where 2-credit sequences 
at the secondary level ate recognized by the state and/or its local eligible recipients. 

State E A secondary student enrolled in a state-approved technical education program who has com- 
pleted 350 hours of the program instruction or who has completed half of the program’s 
required skill assessments. 

State F A secondary student who has earned three (3) or more credits in a single CTE program area 
(e.g., health science or business technology) or two (2) credits in a single CTE program area, 
but only in those programs where 2-credit sequences at the secondary level are recognized by 
the state and/or its local eligible recipients. 

Completer 

OVAE guidance N/A 


State B secondary student who has completed a CTE instructional program. 
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At the postsecondary level, variations in participant definitions across the states ranged from 
relatively minor modifications of the language suggested by the Department—e.g., a student 
who has earned one or more credits in any college credit CTE program or a student enrolled 
with a vocational intent who has earned one or more college-level credits in any CTE pro- 
gram atea—to completely different language based on the number of credits earned or en- 
rollment at a specific point in time (Exhibit C.13). For example, one state defined a 
postsecondary participant as a student who has earned at least six credits required by the de- 
gree or certificate program and is enrolled in at least one three-credit academic or CTE 
course during the fall semester of the reporting year. Another defined a postsecondary par- 
ticipant as either a student enrolled in a preparatory program of study or a student who is en- 
rolled on the 14th day of the freshman fall term in any CTE program; the differences are 
based on type of institution (..e., certificate-granting versus degree-granting). In a third state, 
the participant definition is usually based on enrollment in any vocational course during the 
cohort year, but a different definition is applied to the measure for nontraditional participa- 


tion. 
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Exhibit C.13. 
Postsecondary student definitions 


Participant 


OVAE guidance A postsecondary/adult student who has earned one (1) or more credits in any CTE program atea. 


State B A student enrolled with a vocational intent who has earned one (1) or more college-level credits in 
any CTE program atea. 


State C Definition is based on its use. In most instances, participation is defined by enrollment in any voca- 
tional course during the cohort year. Participation in nontraditional programs is defined as the num- 
ber of CTE concentrators in programs deemed nontraditional for either gender. 


State D A postsecondary/adult student who has earned one (1) or more credits in any college credit CTE 
program. 


State E A postsecondary student who has earned at least six (6) credits required by their degree/certificate 
program and is enrolled in at least one academic or CTE three (3) credit course in their de- 
gree/certificate program during the fall semester of the reporting year. 


State F* A postsecondary/adult student who is enrolled on the 14th day of the freshman fall term in any 
CTE program atea. 


Certificate/ diploma programs: A postsecondary/adult student enrolled in a preparatory program of 
study offered by the regional CTE center. 


Concenttator 


OVAE guidance A postsecondary/adult student who (1) completes at least 12 academic or CTE credits within a 
single program area sequence that is comprised of 12 or more academic and technical credits and 
terminates in the award of an industry-recognized credential, a certificate, or a degree; or (2) com- 
pletes a short-term CTE program sequence of less than 12 credit units that terminates in an indus- 
try-recognized credential, a certificate, or a degree. 


State A A postsecondary student who (1) completes at least 12 academic or CTE credits within a single 
program area sequence that is comprised of 12 or more academic and technical credits and termi- 
nates in the award of an industry-recognized credential, a certificate, or a degree; or (2) completes a 
short-term CTE program sequence of less than 12 credit units that terminates in an industry- 
recognized credential, a certificate, or a degree. 


State B A postsecondary CTE participant who has completed at least 12 CTE credits or completed an in- 
dustry-recognized credential or formal award. 


State C A student who has, within the previous three years, completed a minimum threshold of 12 or more 
units of related coursework in a CTE program area (defined as a two-digit TOP code) with at least 
one of those courses teaching job-specific skills. 


State D A postsecondary student who completes at least one-third of the academic and/or technical hours 
in a college credit CTE program that terminates in the award of an industry-recognized credential, 
certificate, or degree. 


State E A postsecondary student who has completed 50 percent of the credits required for earning the pro- 
gtam’s degree or certificate. 


State F A postsecondary student who is designated as a sophomore, with 30 college-level credits, during the 
fall semester of the cohort year in a defined CTE program of study that terminates in an industry- 
recognized associate’s degree or other award. 


Certificate/ diploma programs: A postsecondaty/adult student enrolled in a TT'C program of study lead- 
ing to an industry-recognized credential, a preparatory certificate, or diploma who completes more 
than 20 percent of scheduled hours in the first term. 
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Exhibit C.13.—cont. 
Postsecondary student definitions 


Completer 
OVAE guidance N/A 
“StateBo A CTE student who has attained a formal award (a degree, certificate, apprenticeship, or an industry 
certification) or completed at least 45 vocational credits with a 2.0 or higher GPA. 
“StateFo A postsecondary student who gains an industry-recognized associate’s degree or other award within 


two years beyond the concentrator cohort year 


Certificate/ diploma programs: A student who demonstrates competencies for a specific job proficiency 
level and receives a preparatory certificate or diploma conferred by the institution. 


* State F maintains separate definitions for postsecondary certificate/diploma programs. 


Similar variations existed in definitions of postsecondary concentrators. Some states adopted 
definitions resembling the language recommended by the Department, but others did not. 
For example, one state defined a postsecondary concentrator as a postsecondary CTE partic- 
ipant who has completed at least 12 CTE credits or has completed an industry-recognized 
credential or formal award. Another state defined a concentrator as a student who, within 
the previous three years, completed a minimum of 12 or more units of related coursework in 
a CTE program area, with at least one of those courses focusing on job-specific skills. Other 
definitions were based on percentage of credits or hours of CTE programs completed—e.g., 
50 percent of the credits required for earning the program’s degree or certificate; at least 
one-third of the academic or technical hours in a college credit CTE program terminating in 
the award of an industry-recognized credential, certificate, or degree; or completion of more 
than 20 percent of scheduled hours in the first term or the number of credits attained. Two 


states also defined postsecondary completers. 


State CTE administrators offered a range of reasons for adopting definitions other than 
those proposed by the Department, and at least one noted that the definitions in his state 
might change in the future. One postsecondary administrator said, “We couldn’t just take the 
concentrator at the 12-credit level. We have completers at 10-credit levels, so we had to fit 
things like that into how our system works.” A colleague acknowledged that compromises 
were necessary to reach agreement on definitions, noting that “for some purposes, we con- 
sider a completer anybody who gets past a certain number of credits. If someone takes 45 
CTE credits within given parameters, we consider that person to have gotten the minimum 
dose.” In another state, a secondary CTE administrator explained, “We didn’t use [the De- 
partment’s] recommended definitions because we don’t use Carnegie units; it was easier for 


us to use hours.” 


One state CTE director described anticipated changes in population definitions and the tea- 


sons for them. He said that the state plans to move from its current secondary definition 
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that pegs concentrator status at completion of 350 hours of program instruction to comple- 
tion of 50 percent of a program, a definition that aligns with one used in several neighboring 
states. He also indicated a longer-range plan “to eliminate the time-in-seat thing, so the as- 
sessments will determine whether your competency level has been attained. Eventually, we’d 
like to rewrite the definition so that completers must pass a state-approved common assess- 
ment, if available.” He explained that these changes are not feasible at present, because “not 


enough of the programs have approved assessments.” 


A local administrator in this state acknowledged that most participants do not turn into con- 
centrators. When asked what percentage of participants do, he responded, “Td like to be 
able to say ... that most of our students who start go on, but if you look at our enrollment, I 
think two-thirds of it is first-year students and one-third is second-year students. From that, 
I’m not sure what you would deduce, but generally speaking ... you could say we have a fair 
amount of attrition.” He attributed some of this attrition to “the natural phenomena” of 
students who sign up thinking they have an interest in the area: “The student might say this 
really wasn’t what I had in mind, and in terms of ability level, some students find out that 
they don’t have the ability to do the program. ... We’re not going to have every single first- 


year student come back as a second-year student. It just doesn’t work that way.” 


There was variation in the extent to which state administrators engaged local representatives 
in developing Perkins population definitions and the effectiveness with which these defini- 
tions were communicated to LEAs and IHEs. An administrator in a state that used the De- 
partment’s guidance for its participant definitions indicated that definitions of secondary and 
postsecondary concentrators “were developed at the beginning of the state plan through our 
accountability committee, so we had a subcommittee with representatives of community col- 
leges and school districts . .. to get their input.” A secondary administrator in another state 
described a similar collaborative effort to build a definition of a CTE concentrator: “It 
wasn’t a definition that we at the state level created all by ourselves. We brought in a team of 
practitioners, CTE directors and college folks, to talk about what the definition of a CTE 
concentrator was. It wasn’t the state level doing what we normally do best as bureaucrats— 
just telling you how to do and this is what it’s going to be.” A local postsecondary adminis- 
trator in a third state noted that the state CTE director “sat in several meetings and worked 
with us around the definitions and the performance measures to make sure that they’re sen- 


sible, that we understand them, and that we can work with them.” 


Local Understanding of Measurement Populations 
Although local Perkins subgrantees are expected to use the population definitions included in 
their state’s five-year Perkins plan, interviews revealed progress toward common definitions 
as well as considerable confusion about and inconsistent understanding of these definitions. 
Some local administrators were quick to acknowledge that steps were being taken to enhance 


the consistency and quality of data. “Definitions and measures ate getting better at the state 
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level,” the director of an area CTE center observed. “We have common definitions now, 
which are the best part. In the past, it seemed that rules would change every year because 
somebody would redefine what the definitions were. ... We do have pathways, and we set 
definitions as we create pathways. It comes down to ‘If you’ve spent a year here, you’re a 


concentrator.” 


Flawed Definitions Can Lead to Inaccuracies in 
Perkins Accountability Reporting 


A practitioner in one local community described how different course-taking schedules 
could affect how concentrators are reported: 


Students take our cosmetology program for four hours a day during their senior year. They 
start before school even starts, they go four hours a day throughout the school year, and 
then they don’t complete until the end of July. That cosmetology program is approved as 
one single CIP code, not two as it says in the definition. | would think those cosmetology 
students . . . would be concentrators, but they’re not . . . according the state definition and 
they do not count in the data.... How they take the courses, what sequence they take the 
courses, what sequence they’re offered makes the difference. Those students, either four 
hours a day one year or two hours a day one year and two hours a day another year, that 
determines if they’re concentrators or not. . . . ’ve been hammering my head against a 
wall trying to convince state administrators that this is a problem. . . . If data are going to 
be pulled and they’re not accurate and not consistent and it negatively reflects upon me, | 
take that personally. 


In another state, local administrators faulted state officials for not doing a good job of com- 
municating definitions. One local CTE director reported, “I don’t think we ever got clear di- 


rections.” A local administrator in another state said: 


Everybody kind of makes up their own definition, as long as they say it’s a 
class that’s going to lead a student into a career pathway. ... We define a 
participant as someone who can be taking the course to continue or just as 
exposure. A concentrator is someone who decided to continue . . . and is 
taking the second or third course in that pathway. .. . So in theory, if stu- 
dents take the second, third, or fourth course within a pathway, they’re 


called concentrators. 


In one state that reported using a collaborative process for establishing definitions, a com- 
munity college administrator noted, “Completers are another big issue. Are they counting 
our degrees as completers, or are they counting as if they meet the gold standard and the in- 
dustry certification as a completer?” A local CTE director in that state suggested that “con- 


centrator” is just a new name for “completer” at the secondary level. 
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We have a senior awards night every year, and I always present certificates 
to the students who were completers. Of course, now we call them concen- 
trators. For the last couple of years, I would say that 75 percent of our sen- 
iors have been completers or concentrators. ... A completer would be 
someone who’s finished the three sequential courses for a career tech pro- 
gram—basically the same thing as a concentrator, [but] they’ve just changed 


the name. 


In another state, both state and local CTE administrators conceded that the definitions were 
not well understood. According to a state postsecondary administrator, the definition of par- 
ticipants is not clear among locals: “You still stumble across people who ask “‘what’s a com- 
pleter?’ even though completers have been out there forever. ... Qut concentrator 
definition is not particularly well known, and it was the best compromise we could make. ... 
If you do the Venn diagram with concentrators and completers, there are a lot of people 
who are concentrators [who] aren’t completers, and that’s expected. But there are almost as 


many people who are completers who would not qualify as concentrators.” 


A colleague at the state level suggested that postsecondary faculty and staff understand how 
to account for the number of credits by course but were uncertain about whether they 
equate credit attainment to being a concentrator. He said that local practitioners understand 
that a concentrator is a student who has enrolled in two or more courses above the explora- 
tory level, but “when we say in the same or related cluster, then how do you count that? 


What about keyboarding because it goes across every single program?” 


Local administrators in this state offered harsh criticism of the population definitions adopt- 
ed by the state and the process used to develop them. One local administrator noted, “It 
would have helped if there had been some consistency in what our definitions were. .. . 
There wasn’t as much collaboration between systems as we should have had earlier on and 
now the definitions have pretty much been mandated to us.” This person also erroneously 
attributed one of the state’s definitions to Perkins. “I assume that we now have a good defini- 
tion of what a completer is because Car/ D. Perkins IV said what a completer is.”* A CTE di- 
rector in another LEA said the state decided to base its definitions on what data are easily 
available—and not on “what the feds are really looking for or what would have to change to 
get accurate data. ... In the definition of concentrator that the state is using, I highlighted 
the word enrolled. It doesn’t say completed, it doesn’t say get any credits, just enrolled. ... 


It’s possible for a student to be identified as a concentrator and never show up for class.” 


? Neither Perkins IV nor the Department’s guidance define (or even mention) completers. 
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Defining Performance Measures 
In addition to suggested definitions of measurement populations, the Department’s non- 
regulatory guidance also includes performance measures that identify the proposed numera- 
tor and denominator for each core indicator. (For example, the suggested numerator for the 
secondary school completion measure is the number of concentrators who earned a diploma 
or equivalent credential during the reporting year, and the suggested denominator is the 
number of concentrators who left secondary education that year.) With the exception of 
nontraditional participation at the secondary and postsecondary levels, all of the measures 
proposed by the Department are designed to evaluate the performance of a state’s concen- 
trator cohort. The proposed nontraditional measures are designed to evaluate participant 
outcomes. Because the recommended measures are based on the Department’s suggested 
population definitions, data comparability is compromised if states have adopted alternate 


definitions of concentrators or participants. 


None of the case study states adopted all of the performance measures recommended by the 
Department. In some instances, changes made at the state level might be construed as rela- 
tively modest—e.g., the insertion of modifiers before or after “concentrators” to refine the 
population included in a particular measure or specification of the assessment instrument 
used for the secondary academic attainment measure. In other instances, the measures 
adopted bore little, if any, resemblance to those suggested by the Department. Four of the 
six states used completers instead of concentrators in at least one of their secondary and 
postsecondary performance measures. Data comparability across states is further decreased 
when states opt to include different groups of students in either the numerator or the de- 
nominator of the measures they adopted. Attachment B provides a comparison of the De- 
partment’s recommended performance measures with those adopted at the secondary and 


postsecondary levels in the six case study states. 


State administrators had varying perspectives on the utility or value of the Department’s 
nonregulatory guidance. Although most reported using some of the Department’s guidance 
to define measurement populations or establish performance measures, some still questioned 
its salience. “We were one of those states . . . that copied and pasted the guidance,” said one 
state CTE administrator, “but there have been some modifications.” A colleague cited the 
value of the guidance in promoting collaboration among the states: “I think that the nonreg- 
ulatory guidance ... has been helpful. I think the spirit continues to grow among the states 


to work closely to adopt or interpret as best we can.” 


In another state, the Perkins coordinator offered a less enthusiastic appraisal of the guidance. 
“T don’t know how helpful it was,” he said. “I guess I would rather have the nonregulatory 

guidance than not have it, but there are issues with it. I have asked [the Department] ‘What’s 
the question you want answered?’ and they can’t tell me. The only problem with the core in- 


dicators is that I don’t think they measure the right things. I think some of the core indica- 
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tors are just giving Congress data, but they’re not necessarily giving them worthwhile data, 
anything they can really use.” A postsecondary colleague was blunter in his assessment of the 
guidance: “I don’t agree with much [the Department] has done. . . . They say it’s nonregula- 
tory guidance, but when it comes to my plan, they won’t approve it unless ’m doing it their 
way.” 

A state CTE administrator in a third state described working with an accountability commit- 
tee to craft measures: “We took all that information and where it made sense we created 
alignment. .. . [O]ne of the guiding principles ... was we didn’t want a measure that was on- 
ly important for the Perkins law. ... We wanted to have relevance to our institutions ... 
within the parameters of what was required from the law.” A counterpart in yet another state 
reported even less use of the guidance: “I didn’t use the nonregulatory guidance. I went to 
[our eligible agency] for my guidance at every level, every step. The negotiations for all of 
our core indicators were done collaboratively between us and our [eligible agency]. I took a 
look at their nonregulatory guidance, but that’s pretty much what I did with it, because there 


really wasn’t much there.” 


State administrators were much more consistent in their opposition to the possibility of 
regulatory guidance (Exhibit C.14). Although state administrators in one state indicated that 
they might support regulatory guidance, their colleagues in the other five states varied in the 


extent to which they opposed this possibility. 
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Exhibit C.14. 
State CTE administrators’ perspectives on possibility of regulatory guidance 


State Would you support regulatory guidance? 


A Probably. I’m a firm believer that the more we can get at common definitions [or] common pro- 
cedures across the country, the better off career tech’s going to be in general. 


Yes and no. Yes in the spirit of trying to create a common comparative approach, but no because 
it typically would mean that it doesn’t come with funds to help implement systems that would get 
us there. 


B With the regulation you have more definition, but then you have less flexibility since each state’s 
data systems are so much different. 


I think if [definitions and measures] had been mandated and there had been regulations... [from] 
Washington...there wouldn’t have been the ownership that we have now [in the state]...Is ours a 
model? Probably not. But do we think it should be regulated? Not even. 


C The reason I don’t want regulatory guidance is the nonregulatory guidance [that] I don’t agree 
with....If ’ve got the same kind of regulations that I have as non-regulatory guidance . . . heaven 
help us 

D Nonregulatory guidance helps us out because regulatory guidance would probably produce more 


work if there wasn’t some local flexibility. 


E I think detailed guidance is better since federal regulations are often not appropriate for shared- 
time systems in career and technical education. 


F [The Department] should have at least nonregulatory guidance for there to be some consistency 
across the states. Without such guidance, it leaves each state on its own to interpret the legislation 
and to decide on how it wants to build its measures. However, the guidance should not be devel- 
oped in a vacuum. [The Department] should convene a committee of state directors to work with 
them in developing the guidance. 


The reasons for selecting different performance measures varied by indicator and by state, 
and none of the six states adopted all of the recommended measures verbatim. Technical 
skill attainment was an indicator for which five of the six states proposed different measures 
at both the secondary and postsecondary levels. For both levels, the Department proposed 
“the number of CTE concentrators who passed technical skill assessments that are aligned 
with industry-recognized standards, if available and appropriate, during the reporting year” 
as the numerator, but several states built in options for use of grades, end-of-course assess- 
ments, hours completed, degree or certificate attainment, CTE curriculum standards, or oth- 
et locally determined methods. A secondary administrator in one state explained: “We felt it 
was better to look at all concentrators and assess them on [whether they met] the gold stand- 
ard as we defined it in our plan... or did they meet it based on the alternative measure, 
which is the teachers certifying that they achieved the competencies based on the curriculum 


frameworks provided to them.” 


In another state, CTE administrators reported four ways in which districts can measure 
technical skill attainment. Alternative measures were deemed essential because only 12 of the 
state’s 9,642 concentrators actually took an industry-based certification exam during the pre- 
vious year. “Not only do we collect whether or not they met it,” a state administrator ex- 
plained, “we collect how they met it. It’s sort of a hierarchy. The highest one is. . . by 
industry certification. The next down is third-party certification, [t.e.,] a NOCTI [National 
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Occupational Competency Testing Institute] exam or Skills USA. The third is whether they 
had a dual credit course for which they received credit... . The fourth and final option is 
what we call ‘locally developed criteria.”’ In this state, several postsecondary institutions were 
positioning themselves to become testing centers for industry certification. These institutions 
were said to believe that part of their goal as community colleges is for students to leave with 
industry certifications. According to state administrators, practitioners at these IHEs envi- 
sion having their students go through the industry assessments on site, which “then gets 
over the barrier of capturing the data results and identifying their success in [these assess- 


ments]or lack thereof.” 


A postsecondary CTE administrator in another state argued vigorously for using GPA to 
document technical skill attainment. As he explained, “The denominator is anybody who’s 
taking a course above the introductory level and the numerator is those with GPA above 2.0 
for those courses. .. . We measure technical skill proficiencies through multiple demonstra- 
tions throughout the semester. Why would I take one test and say that one paper-and-pencil 
test is your level of proficiency? ...So from the perspective of educators in [this state], to 
hang everything on one paper-and-pencil test is not good pedagogy. We feel a grade of C or 


better in those courses is a strong indicator of technical skill attainment.” 


Even the CTE director in the one state using the measures recommended by the Depart- 
ment for secondary and postsecondary technical skill attainment conceded that there are lim- 
itations to the data reported: “We don’t report assessment data to [the Department] for 
pathways where there is not an approved statewide assessment. That’s just something . . . 
we’te going to have to live with until we get valid assessments in all the programs. For the 
others, we try to push for proof of self-reported results. The validity has improved now that 


we've started doing monitoring visits.” 


State CTE staff also flagged concerns about other performance measures—specifically, sec- 
ondary school completion, graduation rates, placement at the secondary and postsecondary 
levels, retention, transfer, and nontraditional participation and completion at both levels. 
Administrators in one state said that they were using the same definition for both secondary 
school completion and graduation “because the law says ‘state recognized certificate or de- 
gree, and the only one this state recognizes is a high school diploma.” In the same state, a 
postsecondary administrator raised concerns about the Department’s guidance regarding the 
retention and transfer indicator: “I don’t like persistence because of the part-time status of 
the community college students. I followed one cohort. Were they there next year? No. They 
were in my leaver cohort. I had 80 percent of the students come back eventually over a 
three- or four-year period. Measuring persistence the way we’re doing it, year-to-year persis- 
tence, is kind of a remnant from the four-year degree model. The median time to a two-year 


degree is seven semesters—and that’s not contiguous semesters.” 
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Another postsecondary administrator suggested that his state might change its measure con- 
structions for retention and transfer, particularly as they relate to timing, to make them more 
useful for internal reporting. He explained that current measures are looking at the cohort of 
students from the previous fall: “We could report [on the cohort] in a consistent way and 
still get at the same figures if we waited and looked at the transfer rate in September. For the 
2009-10 cohort, you would look at the transfer rate in September 2010” (e., at the start of 


the following school year). 


Two states questioned the Department’s recommendations for including “leavers” in the 
measure constructions for placement at the secondary and postsecondary levels. A CTE ad- 
ministrator in one state said, “What we typically do is take completers from the prior year 
and find out if they were placed, rather than all concentrators. .. . It didn’t seem appropriate 
to hold the state accountable for the performance [and placement] of people who had not 
completed the entire program.” A postsecondary administrator in another state identified 
multiple challenges in tracking leavers and conceded that leavers are not a priority for data 
collection: “Only in this round of Perkins have we had to look at placement data for leavers, 
and it’s a real pain in the neck to do that.” He defined leavers as concentrators who left a 


program without completing it and noted two problems with this definition: 


One, some of these leavers come back. Others have left the state, and it’s a 
real effort to track them down. We do a mail survey and follow up by 
phone. Most of them aren’t going to answer a survey if they’re not here. We 
are getting something like a 12 percent response rate for these people, and 
it really depresses our placement rate very significantly. Including leavers 
depresses our placement rate from 88.9 percent for responders to 39.8 per- 
cent. We care more about why people are leaving. The placement rate used 
to be a lot higher when it was for completers, and I think this one now is 
kind of meaningless. We’re asking [the Department] if it’s possible that we 


don’t have to report placement for leavers. 


A postsecondary administrator in another state questioned the wisdom of limiting the time 
frame for measuring placement to a single quarter. “Given the instability of the employment 
data, when we looked at the use of a single specific quarter, especially either October 
through December [as suggested in the nonregulatory guidance] or January through March, 
which are both high and low quarters for retail, agricultural, construction, and a variety of 
other occupational areas, we decided to use a more stable indicator,” he explained. “After 
reexamining numerous configurations for employment ... we determined that the existing 
indicator would insert less confounding variables into the model examining employment 
outcomes.” As a result, the postsecondary placement measure in this state is the percentage 
of CTE program leavers and completers who did not transfer to a two- or four-year institu- 


tion and were found during one of the four quarters following the cohort year in an appren- 
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ticeship program, employment covered by unemployment insurance, the federal govern- 


ment, or the military. 


Administrators in a third state expressed concern about the proposed measures for the non- 
traditional (‘nontrad”) core indicators. As summarized by one administrator, some concerns 
were based on specific aspects of the proposed measures, but others appeared to be more 


philosophical: 


We didn’t like the way they constructed nontrad. If we’re honest, the non- 
regulatory guidance puts a lot of emphasis on nontrad. I don’t think it’s ap- 
propriate to have two separate measures for nontrad. I think we would pre- 
fer that it was just a participation measure. I don’t know any reason to 
believe that the performance of people once they get into nontrad is differ- 
ent from the people who are traditional. ... What we felt was appropriate 
was to make sure the completion rates of people in nontrad programs are 
not different than the completion rates of students who aren’t nontradi- 
tional in that occupation. ... If you want to evaluate that, you could look at 
the difference on the subpopulation indicator [rather than] creating a sepa- 
rate measute for it... . I think it sends the wrong message to have these two 


separate measures. 


Local Understanding of Performance Measures 
There was considerable variation in local understanding and interpretation of the perfor- 
mance measures included in state Perkins plans. In addition, some local practitioners cited 
confusion about how state staff used data to calculate performance. LEAs and IHEs in 
states with more sophisticated data systems did not always exhibit greater understanding of 
these measures or higher confidence in the quality and consistency of data reported. In some 
instances, the lack of consistency in reporting on specific measures (e.g., technical skill at- 


tainment) seemed to be associated with the nature of the documentation required. 


A secondary state CTE administrator in a state that includes CTE data in a consolidated data 
collection to which all districts submit electronically conceded that mistakes were made 
when data were entered into the data system. In this state, a CTE report for each district is 
extracted from the state database and sent to the LEA for validation. As he noted, “Now 
that we’re releasing local performance indicators to school districts, we’ve found that a lot of 
our school districts misidentified their students, miscoded in the area of graduation rate. 
Sometimes ... one student can really make or break a school district, so we’ve been working 
with our districts to help them understand how to identify and code their students.” This 
administrator provided a further description of the problems resulting from the calculation 


of graduation rates for Perkins reporting: 
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Perkins accountability . . . is quite confusing to the school districts. I always 
told school districts to look at local performance indicators as your local 
CTE report card... . What’s really hard is when we go into the Adequate 
Yearly Process (AYP) measures looking at the assessments and the student 
graduation rates. They see the regular AYP graduation rate as being 100 
percent, and then they look at our CTE 481 measure as being 30 percent. 
Of course, by default, they’re in a Perkins improvement plan, because 


they’re not at state target. 


A community college administrator in that state noted a lack of consistency between data at 
the state and local levels: “We just last year finally got to the point where we saw what the 
state was ageregating. There’s a mismatch between what we know on our campuses for 
completion rates and what the state is saying our completion rates are. We’re trying to figure 


out how they are pulling those data. The data are messy at this point.” 


A high school teacher in the same state noted changes in how the state calculates place- 
ments: “We used to do call-ins three years after [the students] graduated. Now it’s all driven 
off their Social Security numbers. We really don’t get the same kind of data we used to get 
when we [did] individual call-ins to talk to students three years later to see what they’re do- 
ing. The state takes care of that, and I’m not sure they share those data. I haven’t seen 


them.” 


A local curriculum director in another state echoed concern about the complexity of inte- 
grating Perkins indicators into state data systems: “It’s complicated .. . what I deal with the 
most is [the state’s] student data website,” he explained. “Coming into this position last year, 
I didn’t really know anything about federal Perkins money. ... Last year was a baptism by fire 
kind of deal in what I had to know. I was a part of all of the name changes that were re- 
quired of classes .. . and trying to align all of that and figure out what kids were concentra- 


tors and what kids were this or that.” 


A community college administrator in another state cited a lack of common understanding 
about measuring technical skill attainment: “There has been such confusion about industry 
certification and what it means. ... For a while, there were two or three lists floating around 
the state... . The state’s saying that we have to report on these industry certifications—we 
have to know who these people are—and not just who they are, but when they took the test, 


did they pass the test, did they retake the test, did they get a job.” 


Secondary administrators in another state also raised concerns about the lack of consistency 
in measuring and reporting technical skill attainment but suggested that the state is taking 
steps to remedy these problems. According to one local CTE director, “The problem with 


data has been that ‘competency’ is subjective to the teacher. Now, we are moving to a rubric 
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that is going to help create more viable or reliable data on that student, but it’s still going to 
be somewhat subjective . . . because it’s still based on what the teacher says at any given 
point. You may go in and I may go in and rate a student differently.” A colleague in this dis- 
trict reported, “We’re still using the same assessment system that we had in place before the 
programs of study. We are still doing the teacher-made tests, and they are part of the courses 
themselves. We have done nothing beyond that. We are working locally and with the state, 


but none of that is in place yet.” 


A postsecondary administrator in another state raised questions about Perkins core indicator 
reporting and whether responsibility rested with the state eligible agency or the federal gov- 
ernment: “Some of the requirements of the reporting that we do for different categories of 
all the statistics just don’t make sense sometimes. It finally occurred to me that they’re not 
sort of standard. ... Seems that the eligible agency is getting instructions from [the Depart- 
ment], and they interpret what they’re hearing. Then they give it to us, and then we’re trying 
to interpret what we’re hearing, and it can be confusing.” She also highlighted the impact of 
seemingly modest changes to the accountability system: “We do what we think we’re sup- 
posed to do, and then the wording gets changed slightly and we have to go back and redo it 


all. It truly is a nightmare.” 


A local CTE director in that state suggested that problems with the state’s measurement of 
technical skill attainment went beyond a lack of consistent understanding or application of 
the measure construction. He said that he had no confidence in the validity and reliability of 
technical skill attainment data. He admitted that he “just provided a number above the state 


target” and guessed that most of his colleagues across the state did the same. 


This director also cited concerns about what happens to data when they are submitted to the 
state: “We submit electronically. [The data are] manipulated behind closed doors, and [they 
come] back out different from what was submitted.” While he said that he had confidence in 
the data collected by individual schools, he was not sure if the data were added together cor- 
rectly at the state level: “Someone at the state office used to send out data and then the state 
CTE director would tell us the data are wrong and to ignore them. You could just look at the 
data and see that they weren’t right, didn’t add up. They were clearly bogus, didn’t pass the 
smell test.”” However, he also indicated that he had seen recent improvements at the state 
level, in part because the state CTE director “has done an awful lot of work on getting high- 
er quality data.” This administrator also noted that definitions and measures are getting bet- 
ter at the state level: “In the past, it seemed that the rules would change every year... . We 
were comparing apples to oranges because how they calculated the data would change. Gen- 


der equity, for example, would change every year, so we were never counting the same kids.” 
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Appropriateness of Perkins IV Core Indicators 
Most administrators and faculty at the state and local levels expressed strong support for 
holding Perkins recipients accountable for attaining positive outcomes and using Perkins 
funds. Some questioned how the accountability systems were designed and implemented, 
and many questioned whether the core indicators are appropriate measures. The indicators 
identified most frequently as inappropriate were the secondary academic attainment 


measures for reading/language arts and mathematics. 


Problems with Perkins Academic Attainment Indicators 


Several state CTE administrators cited reasons why the required measures of academic attain- 
ment fail to provide a valid measure of CTE’s impact on students’ academic achievement. 


The biggest or most obvious [problem] is in academic attainment. In our state, we give an exit 
exam in the 10th grade . . . and the majority of students who enroll in CTE don’t even do so until 
their junior year. In some LEAs, that could go against kids. Why would | enroll a low-performing 
student in CTE when | know two years from now you’re going to look at that measure and hold it 
against me? It really is a nonsensical measure. (State CTE administrator) 


The assessment is administered in October of the 11th grade and measures end-of-10th-grade 
learning, but the students typically begin their CTE core enrollment at the beginning of grade 11. 
So the area CTE centers are held accountable for those scores when that learning occurred be- 
fore they even enrolled in the centers. This has led to huge, contentious discussions. .. . The 
NCLB measurement is being misused in states that deliver shared time systems like this. Use of 
the NCLB measurement is not measuring academic gain; it only measures differences in succes- 
sive cohorts of students. (State CTE director) 


Academic attainment is just a dumb measure. | don’t have any trouble with what we’re trying to 
measure, but the way we’re trying to measure it just doesn’t make any sense. It doesn’t measure 
CTE’s impact on academic attainment at all. (State CTE administrator) 


Our three NCLB measures in this state are dysfunctional. If they were actually binding, they would 
drive us to take on fewer at-risk kids, which is something you don’t want the system to do. (State 
CTE administrator) 


Perkins IV requires states to use the high school assessments administered for ESA for 
these indicators, and state CTE directors reported that the Department offered no flexibility 
to use alternative assessments. The ESEA assessments were widely viewed as inappropriate 
because they typically are administered in the 10th grade or early in the 11th grade, and stu- 


dents are more likely to take CTE courses as juniors and seniors. 


Administrators in most states questioned the emphasis placed on the nontraditional indica- 
tors and the wisdom of holding districts accountable for outcomes that are difficult to con- 
trol. As one noted, “We have two or four nontraditional measures; the majority of our 


measures are things that are dysfunctional.” A colleague concurred, “The feedback I get 
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from deans is that there are so many factors out of our control... . If ’m at a technical col- 
lege, I have a lot more of the hard trades . . . [and] probably a lot less of the nontraditional 


gender students.” 


A secondary administrator in another state said, “One that I really question is nontraditional 
patticipation, particularly for the smaller agencies. It’s possible that they don’t have any non- 
traditional programs. By law, we can’t require them to operate programs unless we can pay 

for them, which we can’t do. So they’re kind of destined to be sanctioned down the road be- 


cause of something that they can’t do, and they don’t have the resources to do.” 


In a third state, the executive director of the statewide CTE council said, “We have a huge 
issue with nontraditional. There’s just no way you can hog-tie kids and put them in classes 
they don’t want. That’s a big issue with local directors.” Another state CTE director said, 
“The ones I question the most are .. . the emphasis [on] the nontraditional. To be held ac- 
countable in the same way there as for performance in math and reading doesn’t feel right to 
me.” 

A local administrator in another state explained how the nontraditional indicators put pres- 


sure on him to spend Perkins funds in different ways: 


I haven’t in the past had a lot of money for nontraditional, but now because 
of the accountability data, my nontraditional data look like they need im- 
provement—as they do for every other center. So because of the accounta- 
bility data, we are spending money on nontraditional stuff like promotion. 
We can keep measuring it. I don’t think anything is going to change. ... 
People are going to sign up for what they want to sign up for, and that’s 
just a hard nut to crack. In auto body, females do sign up for it, but they'll 
never sign up in the same proportions that males do. Same in aviation. 


There are some fields that just will never be 50/50. 


Others voiced concerns about how postsecondary populations are defined and how reten- 
tion is measured. “I don’t think it’s irrational to set targets and hold people accountable to 
them,” a state postsecondary administrator explained. “I think it’s irrational to say 100 per- 
cent of all students entering your institution will be in each measure. ... I don’t know who 
came up with that.” He noted that the “accountability mechanisms are out of sync” and sug- 
gested that “a lot of the accountability is really predicated on the old four-year model of an 
18-year- old newbie who comes into the community college, gets out in two years, and then 
goes on to a four-year [institution]. He also indicated that he liked “the leverage that ac- 
countability has given the local deans.” One of his colleagues questioned the feasibility of the 


retention measure, noting that “We shouldn’t be focusing on retention or persistence in a 
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Data 


Data 


program when the economy has changed to make that program less viable. Students ‘vote 


with their feet’ in enrollments.” 


A workforce development administrator in another state echoed concerns about the wisdom 
of measuring retention at the postsecondary level. He reported, “Our biggest problem is a 
conceptual one—the issue of what do you consider a completer. It’s a baccalaureate degree 
orientation or high school graduation orientation. The big chunk of what happens in work- 
force education at the community college level is on the ability-to-benefit model; it’s not to 


get a degree.” 


Collection and Reporting 


CTE administrators, faculty, and staff appear to be taking the strengthened Perkins IV ac- 
countability provisions seriously and moving toward systematic and rigorous approaches to 
data collection and significant emphasis on data quality. While Perkins accountability systems 
in all six states could still be described as works in progress to varying degrees, state CTE 
administrators exhibited a thorough understanding of the Perkins accountability requirements 
and a commitment to helping LEAs and IHEs collect valid and reliable data. Most local re- 
spondents could identify how the state had assisted with Perkins IV data collection, but their 
reviews of state assistance were not universally positive. Even in states with web-based re- 
porting systems or longitudinal tracking capacity, some Perkins recipients reported frustration 


with reporting processes and information provided by state administrators. 


Noticeable differences in accountability systems were apparent in the two states visited for 
both the 2004 National Assessment of Vocational Education (NAVE) funding and account- 
ability study and the current study. State administrators in both states had pushed for greater 
consistency in collection of Perkins data, deeper local understanding of the purposes and im- 
portance of Perkins reporting, and new approaches to assessment of technical skills. When 
visited in 2002 for the NAVE study, both states still relied heavily on teacher-developed 
checklists for documenting technical skill attainment. By early 2010, one was rolling out a se- 
ties of third-party assessments, developed with state and local input but administered and 
managed by an external vendor, and the other was developing a competency-based assess- 
ment rubric that includes standards in five areas: knowledge attainment; technical skills; 


problem solving; career awareness; and communications/literacy. 


Systems and Reporting Capacity 


All six states were making use of technology to streamline data collection and reporting and 
increase consistency. Each had established some type of electronic reporting system, but the 
level of sophistication varied dramatically. Only one of the states had a comprehensive longi- 


tudinal data system that combines education and workforce data, including unemployment 
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insurance wage records, to track outcomes of CTE programs. Different local entities in this 
state were responsible for uploading data into the different systems; at the state level, data 
could be drawn and combined for different purposes (reporting, research, funding, etc.). The 
Perkins director stated, “We're able, with all of our longitudinal data, to hone in on how our 
students are doing, how our programs are doing, how individual programs are doing, so that 


they’re very accountable.” 


At the other end of the spectrum, the CTE director in another state acknowledged that data 
had not been valued in the past. He noted with some chagrin that “our data system is literally 
sitting under an employee’s desk out there on a PC on an Access data base that her son de- 


signed years ago... . The data system architecture is a huge question in my mind.” 


Two states had integrated student-level databases linking Perkins data with information from 
other academic programs, and the other three had plans to implement more comprehensive 
data systems. In one of these states, a state data administrator described efforts to link its 
CTE reporting system with other student-level data in its electronic information system. He 
expressed confidence that “a lot of Perkins reporting and accountability issues will go away 
and data will get much better,” as the state invests in a longitudinal data system. He also not- 


ed the critical importance of coordinating with postsecondary partners: 


The state’s working right now on a database, if we can ever get all our post- 
secondary [institutions] to cooperate with us. Every student [who] goes to 
public school in [this state] has a state ID number—not the Social [Security 
number], but an assigned unique number. If we can just get .. . all the dif- 
ferent postsecondary institutions to agree [that] everybody uses the same 
number, then we can simply follow those students. ... At this point, we 
have a unique number, the [technical college system] has a different num- 
ber, the community college has different number, and the [university sys- 


tem] has a different system, so we’re trying to get everybody on the same 


page. 


The Perkins coordinator in another state explained that an ambitious student-level database 
had been stalled by the state’s economic downturn, but the CTE director was ready to esti- 
mate a time frame for the project. He said, “Within three years, we should have a system 
that’s starting to crank out relatively good data. We’ve put in a proposal for funding to estab- 
lish a longitudinal database across all the levels of higher education and K—12, as well as do 
data matching with the employment development department. .. . Right now the barriers are 
FERPA and the Social Security issue. There is absolutely no willingness to use SSNs as stu- 


dent identifiers.” 
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The CTE director in a third state was optimistic about unveiling a more limited system by 
fall 2010: “We are going toward a statewide grant management system that will have online 
Perkins applications [and] online monitoring document{s], and the original design we’ve 
talked about would also carry a parallel data gathering form. ... Down the road, [I'll proba- 
bly have to pay for design of] the web-based data collection that would be on the statewide 


grant management system.” 


Changes in Data Collection and Reporting for Perkins IV 
Many case study respondents described data system improvements they had implemented in 
response to Perkins IV requirements. One state CTE director reflected upon how far his 
state had come with regard to data quality in a short time: “We used to refer to [our state] as 
kind of a data desert. We’ve changed dramatically in the last five years, and we’re getting bet- 
ter all the time. We created a portal where all schools go in and register their programs of 
study, and then we approve them on that electronic board... . [We’re] not as strong on out- 
comes. Our second round of state longitudinal data systems grant that we applied for would 
provide a lot more.” The deputy director provided more details about the new data system: 
“Until 07—08, [our state] only reported aggregate data at the state level; that was the first year 
the state began reporting school-level data. One of the big projects of our state longitudinal 
data system grant is a decision-support system, where you’re able to mine this big monster 
data queue and go in and look at data and slice and dice it however you would like. .. .We 
put a bunch of our eggs in that particular basket as a vehicle for us to get reports, trend anal- 


ysis, and drill-down capacity of identifying special populations.” 


State CTE administrators in another state reported that their quality assurance process im- 

proved between Perkins III and Perkins IV, citing more local visits for technical assistance on 
the state’s management information system. The assessment coordinator for this state noted, 
“Transitioning from Perkins III to IV produced a lot of work for a lot of MIS people because 
they may have had to reconfigure any local reports. ... But we just tried to rely, wherever we 


[could], on something that’s already being collected.” 


A state Perkins administrator in a third state suggested that the Perkins IV changes to ac- 
countability requirements had greater impact at the secondary level. “Perkins IV was nothing 
new to the college system,” she remarked. “[With] Perkins III at the secondary level, they re- 
ported their data. Sometimes it was in a timely manner, and other times it might not have 
been.” A postsecondary colleague elaborated further on the Perkins IV changes: “In [Perkins] 
UI, 1 think...we had a lot of focus on [on nontraditional and] bringing those numbers up. 
[Now] they’re looking at all of the accountability measures. They’re making relationships be- 
tween student achievement initiatives.” Administrators in this state also noted changes in 
how Perkins data are shared with postsecondary institutions. Under Perkins ILL, state officials 
shared only aggregate data; now they get individual performance reports that they can com- 


pate to the aggregate performance indicators for all community and technical colleges. 
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Data Reporting Roles and Responsibilities 

As noted above, all of the case study states had at least some capacity for electronic report- 
ing of Perkins core indicator data, but that capacity ranged from sophisticated web-based 
portals to more primitive systems in which eligible agencies submitted data via e-mail or CD. 
The more sophisticated systems were designed to support aggregation and initial analyses of 
Perkins data with little to no additional effort at the local level. The less sophisticated systems 
required more extensive input from LEAs and IHEs. As a general rule, postsecondary data 
collection and reporting systems were less likely than their secondary counterparts to rely on 


manual tabulation or input of student-level data. 


Although states have made significant investments and efforts, LEAs and area CTE centers 
are still expected to assume primary roles in the collection or validation of student-level data 
in most states. Specific responsibilities for secondary data collection and reporting varied 


actoss the case study states (Exhibit C.15). 
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Exhibit C.15. 


State and local responsibilities for secondary Perkins data collection and reporting 


LEA. 


State staff compile data and calculate state and local per- 
formance on cote indicators; academic attainment data 
ate added at the state level. 


State staff obtain technical skill assessment data from 
website of third party vendor. 


State staff disaggregate demographic and special popula- 
tions data for CTE participants and concentrators to 
examine differences in performance. 


State State responsibilities Local responsibilities 

A Electronic portal is used for data collection on concen- School districts pull data from their student information 
trators, participants, and special populations. systems and populate the program fact template, which 
Unique ID for each student allows state staff to disagere- includes every participant, with program code and level 
gate data to obtain an unduplicated count. of participation. 

Statewide contract with the National Student Clearing- Districts upload data to the department of education 
house provides access to postsecondary placement data. portal. 

State staff members generate a verification report speci- Local staff complete the post-school survey to feport 
fying which concenttators graduated and ask local staff whether graduates weneon tO postsecondary education, 
to identify what those students were doing the second employment, or military service and submit those data to 
quarter after the year in which they graduated. the portal. 

State staff access and compile placement data. 

B New electronic secondary data system is used to collect Local responsibility for data submission and validation 
grade history, credits, and state course codes mapped to may rest with CTE director, data system coordinator, or 
CIP codes. superintendent. 

CTE data are part of consolidated student data report Teachers are expected to verify student information prior 
required of all districts; a vocational report is extracted at to submission. 

the state level for validation by local staff. LEA staff report completer information to the state 
Concentrators ate identified at the state level based on database. 

information submitted through the new data system. 

C The department of education collects data on secondary Districts pull data from their student information systems 
core indicators from districts through a separate online and hand-enter data on enrollments into the state system. 
data collection system. Some districts use software that pulls demographic data 
Enrollment data collected in the fall include de- as well as participant and concentrator information from 
mographics, participants, and concentrators from the the school or LEA’s student information system. 
previous year; placement data are collected in the spring. — some larger districts employ dedicated data staff; in 
The department of education aggregates statewide data smaller districts, CTE and counseling staff often handle 
and examines data at the LEA level to identify those reporting. 

LEAs that are those underperforming. 

D The state has a comprehensive, longitudinal data system Raw data are extracted from the LEA’s data elements 
with standardized definitions and guidelines for each data and sent to the department of education. 
element in the local management information system. LEA staff review reports generated at the state level and 
Department of education staff receive raw data from make corrections as needed. 

LEAs and calculate accountability measutes (e.g., con- Local CTE directors use an online system to establish 
centrator) using algorithms. local performance levels. 
The state also has a workforce data system that can de- 
termine placement information. 
State staff provide each LEA with an aggregated report 
on its performance on Perkins indicators. 
E Department of education staff e-mail a zip file to the School staff member collects data from individual teach- 


ers, who typically provide information on paper rather 
than electronically. 


Teachers identify participants and concentrators and 
students who are members of special populations; coun- 
selors or teachers at area CTE centers may have to call 
the sending schools for student information. 


Area CTE centers have primary responsibility for collect- 
ing follow-up data for the placement indicator. 


Information is imported into the state database format 
and put on a CD that is sent to the state. 
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Exhibit C.15.—cont. 
State and local responsibilities for secondary Perkins data collection and reporting 
State State responsibilities Local responsibilities 
F The state uses a web-based data collection system to CTE directors and teachers report their data electronical- 
capture CTE class data, student concentrator status, ly though an upload function on the state agency’s web- 


competency information, dual credit and dual enrollment site. 
data, and follow-up data on concentrators. 


and information related to dual enrollment courses. 


Some teachers enter their data and upload attendance 
figures through a statewide student information man- 
agement system used by districts on a voluntary basis. 


At the secondary level, four states had web-based reporting applications that eliminated the 
need for LEAs to submit redundant student profile data and helped identify concentrators 
and completers. The other two states employed a two-tiered data reporting strategy that re- 
quired LEA staff to assemble data from a variety of sources and package the data for sub- 
mission to the state. In both of these states, concentrators were identified at the local level. 
Only one state claimed it had the capacity to generate placement data at the secondary level. 
In four of the other states, LEAs had primary responsibility for collecting placement data; in 
the fifth state, the eligible agency engaged a contractor to perform this function. A secondary 
administrator in this state described the contract for collecting secondary placement data: 
“We contract placement out with a state university, and we’re going to continue to use a 
contractor until we get our statewide longitudinal data all worked up. [The contractor] does a 
lot of work for us. They do UI wage matches for us, FEDES matches, [and] military wages. 
They go through the National Student Clearinghouse to find our students at universities and 
colleges throughout the nation through SAT matches and ACT matches as well.” 


Although postsecondary data systems were frequently described as more sophisticated than 
their secondary counterparts, this was not always the case (Exhibit C.16). For example, one 
state-level administrator noted: “The other big challenge for us was on the postsecondary 
side. It [the postsecondary data system] doesn’t exist, so what we’re doing now is having the 
[postsecondary institutions] provide us aggregate data that they have... . I think we’ve made 
inroads in creating some data quality, but... either the system doesn’t have the capacity, as 
far as its technology infrastructure, or the program people have a hard time communicating 
what data they actually need to the information technology data analyst people. There’s just a 


disconnect.” 


Teachers are expected to input competency assessments 
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Exhibit C.16. 
State and local responsibilities for postsecondary Perkins data collection and reporting 


State State responsibilities Local responsibilities 

A State staff created a manual to guide extraction and Postsecondary institutions compile, extract, and submit 
compilation of postsecondary Perkins data. performance data set forth in the state data manual. 
State staff provide technical assistance to IHEs through —_ Data typically are compiled by the institutional grants 
meetings with all Perkins-eligible IHEs, webinars, and office, the institutional advancement ot research office, 
one-on-one support. or others and approved by the college president or 
At present, there is no state data system for postsecond- CEO. 
aty Perkins information. 

B The state has a common data system and standardized Each college houses its own database, in a system com- 


extract used by all postsecondary institutions. 


A negotiated process (i.e., one-time exception or perma- 
nent expansion of the extract) occurs whenever state 
staff request data that are not part of the standardized 
extract. 


Information is compiled at the state level before going 
to the data warehouse 


All community colleges have a unified database [a 
statewide database into which both secondary and post- 
secondary systems can load information] from which 
state staff extract data. 


The state provides guidance to locals on data confidenti- 
ality. 

Staff in the state postsecondary CTE office conduct data 
analyses and provide colleges with program-level re- 


The state has a comprehensive, longitudinal data system 
with standardized definitions and guidelines for each 
data element in the local management information sys- 
tem. 


Department of education staff receive raw data from 
THEs and calculate accountability measures (e.g., con- 
centrator) using algorithms. 


The state also has a workforce data system that can de- 
termine placement information. 


State staff provide each [HE with an aggregated report 
on its performance on Perkins indicators. 


State staff met with staff at IHEs to discuss definitions 
and performance measures. 


State staff compile data and calculate performance on 
cote indicators. 


The state has a common data system used by all post- 
secondary institutions to collect and report Perkins data. 


State staff trained local student services personnel on 
data input and common definitions. 


State staff compile data submitted by individual IHEs 
and centers and prepare reports. 


mon throughout state. IHEs use a standardized extract 
to submit data to state CTE staff. 


Staff at individual IHEs are responsible for coding intent 
and exit codes, which are uploaded into centralized sys- 
tem quarterly. 


Data sent to the state include information on enroll- 
ments and grades and unit records 


Community colleges conduct supplemental data collec- 
tions at registration or administer classtoom surveys to 
obtain special populations data. 


Many colleges started using administrative data matches 
instead of follow-up surveys, which were reported to be 
time consuming. 


IHEs send raw data to the state for calculations of the 
measures. 


Performance reports received from the state are not 
disaggregated at the program level. 


IHEs submit separate Perkins reports to the eligible 
agency; most data are harvested from the data ware- 
house or system at the individual IHEs. 


Administrators identify participants and concentrators, 
with assistance from program managers and faculty. 
Placement data are obtained through surveys adminis- 


tered by staff responsible for tracking all placements 
(not just those of CTE students). 


Local staff enter data into the state system using pre- 
scribed common definitions. 


IHEs receive reports on their performance on the core 
indicators. 
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In contrast, a postsecondary state administrator in another state noted that all community 
colleges have a unified database and said, “Unfortunately, K—12 is not in that place. They 
have, in my understanding, five pretty common platforms. ... Beyond those five, there are 
other homegrown systems, hand record systems still, in some cases.” In a third state, a work- 
force development researcher gave high marks to the postsecondary data system: “The post- 
secondary system has been in place for quite some time. This is one place where the 
postsecondary is much more of a system than the secondary. There is a single database de- 
sign and software. The databases actually reside at the institutions, but there are standardized 
extracts that come pumped up to the state office. They’ve recently increased what’s in those, 
but there’s a negotiated process every time you want to get data that aren’t in the standard- 


ized extract. ... It’s sort of an industrial-strength management information system.” 


State Technical Assistance and Support 
State administrators in all of the case study states reported offering technical assistance on 
Perkins accountability measures and reporting procedures. The most frequently cited forms 
of technical assistance were workshops or some type of written instructions. In addition, 
most administrators described a process by which local practitioners could access additional 
help from state staff via telephone or e-mail—and many local practitioners indicated that 


they had taken advantage of this customized assistance. 


One state CTE director reported a broader partnership across the state education agency to 
support data collection, noting that Perkins accountability reporting is now included in 
statewide data summits and data training. He also said that the CTE division offered its own 
training. His deputy indicated that state staffing for data-related services includes five full- 
time help desk attendants who respond to “questions associated with data uploads, data 
connections, and data validation” and four regional trainers who “work with school districts 
all year long” and conduct a series of regional “data input parties.” Because these staff re- 
spond to questions related to multiple federal programs, he explained that their time is 
charged to specific programs depending on the nature of the call or inquiry. He also reported 
that the state education agency sponsors an annual data quality conference attended by ap- 
proximately 700 district staff, with program-specific breakout sessions that allow CTE staff 
to focus on topics such as the post-school survey and how to count students who transfer 


from other schools. 


However, a high school principal in this state suggested that there was still room for im- 
provement in the state’s technical assistance delivery, particularly for LEAs using Mac com- 
puters for data collection and reporting. “They tell me all the time it’s just the loose nut on 
your keyboard,” he reported. “I agreed sometimes. It takes quite a little extra time and 
help—and they do have good help available on the help line. ... After a fashion, it all gets 
worked out and it gets up and rolling again.” A local administrator in another district was far 


mote Critical: 
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[O]ur state doesn’t have a really good data warehouse and a good, current 
functional database for me to pull from... . It’s the most dysfunctional sys- 
tem I’ve ever witnessed, and I’ve worked in six states... . When I have to 
go through. . . the portal and get the data. . to fill out the report, it’s just an 
absolute nightmare. The help desk is now dial-a-prayer; there’s nobody 
there to ask for help. If you put in punctuation other than commas and pe- 
tiods in the portal, it rejects your answer. Once you learn that you can’t do 
all that stuff, it’s time consuming. . . to take any documents you’re trying to 


download and take all the punctuation out of [them]. 


In the state with the most sophisticated longitudinal data system, state CTE administrators 
reported that division staff members “spend a lot of time on technical assistance for input- 
ting data so everything is standardized. There are definitions and guidelines for each data el- 
ement in the local management information system so data submitted to the state are 
accurate.” The state CTE director pointed out that the technical assistance extends beyond 
guidance on input to identification of priorities for program improvement. “We have a strat- 
egy where we work with locals to develop their plans,” she explained. “[We ask] what’s the 


focus of that year in terms of the indicators? ... At some point you have to focus.” 


A third state administrator stressed the importance of using data: “State research and data 
staff do local workshops not only on how to input data but also on how to look at data for 
program improvement.” In addition, she noted peer support for Perkins accountability re- 
porting: “There’s a community out there among LEAs. Many of them talk to each other 
about how they’re doing. If one LEA is struggling, I think they tend to want to find some- 
body who isn’t and work with them. Thete is really a strong CTE community among the di- 
rectors on the secondary and postsecondary sides where they seek solutions from each other. 
... The big providers and the smaller providers, we’re really seeing commonalities in terms 


of their strategies, and I think that’s been really successful.” 


In yet another state, both secondary and postsecondary CTE administrators described the 
state’s technical assistance efforts: “Technical assistance workshops were probably the big 
ah-ha,” according to a postsecondary administrator. “We’ve been mentioning these measures 
since we first learned about Perkins IV... . It really didn’t become real until we had this in- 
formation out and really got to do some of the more personal discussions. We’ve [done] 
presentations. ... We’ve had [the information] published on our website for them... . We’ve 
distributed it along with the [negotiated state performance levels]. Another administrator 
described the customized assistance provided at the local level if students are not properly 
coded or counted: “We would go one by one, record by record, with the school districts .. . 
to see where they were. That’s technical assistance down to the fine grains of sand, and it’s 


an excellent opportunity for them to see the importance of accurate coding.” 
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A secondary administrator in this state reported receiving phone calls from superintendents 
who asked for explanations of core indicators. She said some admitted that CTE hadn’t been 
a priority in the past but indicated that it would be in the future if it affects funding. A col- 
league concurred, noting that “local CTE directors wanted information to be shared with 
superintendents when we told them that they would be held accountable for performance 


data.” 


Nonetheless, officials in this state acknowledged that their technical assistance efforts were 
not universally valued at the local level. One commented that local practitioners’ perceptions 
were “they don’t want to help us, they don’t care,” an observation confirmed by some local 
staff. One local practitioner suggested that state staff could be a barrier to moving forward 
on data analysis. She described plans for a summer retreat for a task force from the directors’ 
association to look at the data that were collected up to that point. “The state encouraged us 
to cancel that meeting” because proper data were not available, she explained. “So it wasn’t 
until December that they finally got the data-sharing agreement in place, shared appropriate 
data, and pulled information at the state level. But truly, July to Christmastime, they couldn’t 
give us any answers.” A counterpart in another district reported “a general sense of frustra- 


tion in terms of lack of direction and guidance from the state.” 


State administrators in two other states indicated that workshops were their primary forms 
of technical assistance. “We do an August conference that is aligned around all of the Perkins 
accountabilities, assessment development using data to inform instruction, and integrated ac- 
ademics,” one state CTE director reported. “Then we sponsor regional or other statewide 
pieces throughout the year to reinforce some of those things.” This administrator also indi- 
cated that he was encouraging local subgrantees to contribute to these efforts “so that we'll 
have that buy-in, and then we'll pick up the balance of that just so that they feel that’s part .. 
. of their responsibility.” However, an IHE administrator in this state suggested that state 
Perkins officials could have provided more technical support: “The main thing they’ve done 
is sat in several meetings and worked with us around the definitions and performance 
measutes to make sure they’re sensible, that we understand them, and that we can work with 


them. Other than that, there’s been very little advance on the technical front.” 


In yet another state, a secondary administrator described a series of workshops and webinars 
offered to assist local Perkins subgrantees with accountability requirements. “That’s one of 
the reasons I’ve designated one of my consultants as being the data guy,” he reported. “It’s 
his responsibility to make those kinds of things happen.” A local Perkins coordinator said 
that help was available at the state level if she called: “The consultants [at the state level] di- 
vide themselves up . . . and each region has its own person to work with. He’s usually really 
good if I send him an email. He’ll make sure that he emails me the same day, but it’s basically 
questions like, would this be allowed? .. . Or how do I figure out what my core indicators 


ate?” 
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Local Data Collection Efforts 
Case study visits identified extensive local efforts to collect and report valid and reliable Per- 
kins accountability data. At both the secondary and postsecondary levels, respondents were 
able to identify specific actions they had taken to improve data quality and respond mote ef- 
ficiently to data (or data validation) requests from state Perkins administrators. Most were 
confident that they had submitted accurate data for most indicators. In addition, many also 
noted that their data collection efforts were constrained by shortages of time and re- 
sources—human, financial, and technological. Several local practitioners suggested a need 
for better alignment of state and local reporting efforts. Some also suggested that the Perkins 


indicators included some information they would be tracking anyway. 


In the state with the most sophisticated longitudinal data system, one local secondary CTE 
administrator said, “Perkins data are collected through our MIS system, which submits the 
report directly to the state... . We have a data processor at each of the high schools. .. . 
[W]hen the guidance counselors do their schedules. . . the data processors put that infor- 
mation into the system ... I can go in and see who’s in each of the CTE programs and also 
see their sequence of courses.” After proudly noting that her district had the top graduation 
rate in the state, she said that district administrators would be tracking this information re- 
gardless of Perkins requirements: “Graduation rate . . . it’s something that we are on top of 
for all students. . . . If we didn’t have to report it for Perkins, we'd still be on top of it.” A 
faculty member in another LEA reported significant local involvement in the collection of 
placement and retention data. “We have to follow up on every single one of our students 
who come through the program, whether they complete or not,” she indicated. “[The report] 
goes to the state... but some of the information we get back from the state is different than 


what we have here. I don’t know where they are getting [those data].” 


Local postsecondary administrators in this state also reported significant involvement in col- 
lection of placement data and described efforts to integrate Perkins reporting into their insti- 
tutional systems. One IHE instructor noted that “it’s pretty easy” to get follow-up 
information on former CTE students: “We’ve got a small program, and students notify us or 
employers notify us.” He said that most of the students work in the area, so he typically re- 
ceives reference calls from employers or students stop by to tell the staff when they get a 
job. “Then we’ve got an employer and a graduate that we'll follow with a survey 6 to 12 


months post-graduation,” he stated. 


An administrator at another IHE in this state reported the capacity to track students into 
baccalaureate institutions “if they went to one of our local university partners.” She said, 
“We know that because of a local arrangement, not because there is a statewide system that 
provides that information.” This administrator also noted that the college had incorporated 
Perkins data into the “data dashboard model” used for all programs: “When we’re dealing 


with external funding agencies and they have different requirements or things that add to 
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what we’re doing for the purposes of program viability, health, and institutional effective- 
ness, we find that it’s best to try to incorporate [those requirements] rather than create a sep- 
arate system.” Noting that concentrators are the denominator for a number of Perkins 
measures, she indicated that “we began to pull that information into what we do institution- 
wide.” 

In another state, local administrators reported hiring new staff or training current staff to en- 
ter data in the state’s new web-based data system. In one district, the CTE director said a 
designated staff member was hired to enter data into the Perkins IV portal and noted that 
teachers were required to code the data. Another administrator in the region noted that 
some smaller LEAs did not have staff specifically responsible for Perkins reporting, so the 
regional intermediary agency provided workshops that trained staff to enter the necessary 
data. A principal in another LEA indicated that Perkins data “are all set in our computer; I 
can just say ‘run the Perkins information.”’ A colleague noted that the introduction of the 
new system made reporting far less tedious and enabled administrators to transfer most data- 
entry responsibilities to a secretary. “The data are basically a function of what classes the kids 
are enrolled in, and that’s already on the system,” she remarked. “We just have to code the 
classes as to which career track they belonged to.” However, the CTE director in one of the 
state’s largest LEAs questioned the “artificial distinction” of separating Perkins data from 
other information in the local data system. “We want Perkins funding to continue,” she said. 
“But evaluating Perkins by itself in our district is not really necessary because it’s not different 


in our district.” 


In another state with a web-based reporting system, local practitioners described labor- 
intensive efforts to collect Perkins accountability data. One local CTE director reported a 
creative strategy for getting assistance with the collection of follow-up data, which is done by 
phone and mail surveys. He said that he asked the LEA’s human resources department to 
identify a teacher who had been placed on administrative leave. He explained that rather 
than having this person “go home and sit and draw a check,” he could put the teacher to 
work inputting data. “So it’s not a cost to me, and it’s actually putting somebody to work 


[who] needs to be working,” he explained. 


In one of the state’s largest districts, the CTE director reported hiring a team on a part-time 
basis to help look at the data: “We bring in a team of business teachers and other teachers 
who have done this before; they go school-by-school and student-by-student to determine if 
these data are correct.” He said that the “state determines the number of students we ate go- 
ing to do follow-ups on—and we have a team that comes in here and does nothing but fol- 
low-ups.” His counterpart in a much smaller district indicated that mail surveys had typically 
generated about a 30 percent response rate, so he devised a new way to get more complete 
placement data. “When they come back to get their senior yearbooks, my assistant sits there 


with the follow-up surveys,” he reported. “They usually come back in August before they’ve 
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gone off to college, and every student is going to come back to get that book. That is the 
most up-to-date, accurate information. ... [With] the yearbooks, we get about 95 percent. 
We also go into the senior English classes to ask them about the students who graduated last 


year.” 


Local administrators in a state with a less sophisticated data system emphasized the time- 
consuming processes required to assemble and report data. While not required to do so, sev- 
eral indicated that they had purchased Grant Link software to help organize the required in- 
formation. “Grant Link kind of organizes it in the way that ’m going to have to manually 
input everything on the state website,” one local Perkins coordinator observed. “It usually 
takes about a month and a half from the time that I get the information and I make sure that 
everything is accurate on the software. Once the mapping portion of it is correct, then every- 
thing else is easier because it’s a matter of just pushing a button to create whatever report I 
need.” In another district, the person responsible for data said that the “software generates 
the numbers and grabs the student data,” but noted that considerable set up is required: 
“The set up includes ... making the information from our student information system avail- 
able to the software, marking the courses that are... CTE courses, and marking the gradu- 
ates—the 12th-graders who actually graduated because we’re doing this a year in arrears.” In 
another district in this state, an administrative assistant was compiling Perkins data without 
third-party software and demonstrated the cumbersome “cutting and pasting” that was re- 


quired to enter Perkins data through the state’s online data collection system. 


Challenges 
At both state and local levels, respondents identified several challenges or barriers to their ef- 
forts to produce valid and reliable Perkins data. Some were related to the process or mechan- 
ics of data collection, while others were based on the content of the information required to 
report on the core indicators. Some respondents identified both process and the content as 


problematic. 


State officials and local practitioners in most states identified myriad challenges associated 
with the collection of technical skills assessment data at both the secondary and postsecond- 
ary levels. Beyond the challenges associated with the lack of widely available, affordable 
technical skills assessments at the secondary level, as previously noted, officials also identi- 
fied challenges in obtaining the results of state or nationally recognized assessments. ““That’s 
a huge challenge,” one state secondary administrator said. “Once students graduate, they’re 
not required to come back to tell their teachers ‘I received my cosmetology license or my RN 
license or whatever.’ So our school districts don’t report on that—or only report on this 
handful of students that do come back to . . . show their certificates to their instructors.” A 
colleague pointed out that students who come back to show off their certifications are not 


part of the cohort for the current reporting year. 
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In another state, local postsecondary administrators expressed similar frustration about the 
heavy reliance on industry certification as the preferred measure of technical skill attainment. 
One administrator noted challenges in gaining access to certification examination results: 
“The board of nursing will give us that information but it will cost a lot of money; we will 
get the information when we pay the $2,000 or $3,000.” In contrast, an instructor said the 
college doesn’t have to pay for results from the emergency medical technician examinations: 
“We can just call the state and we get it. We don’t get it very quickly, but we can get it.” Ear- 
ly childhood education credentials were considered particularly problematic at this IHE, be- 
cause alumni “have five years after they get the degree to take the certification exam, and we 


don’t even know if they do it or not.” 


A counterpart at another college noted similar concerns. “If industry certifications are going 
to be retained as part of the measure, then there has to be a response for collecting [them],” 
she stated. “I think that we would all be reported as underperforming simply because of the 
complexity of getting the data... . It just doesn’t make sense for [each of the] community 
colleges to have to contact the nursing licensing board for their individual population of stu- 
dents’ pass rates.” She also recognized the privacy considerations associated with access to 
certification exam results and suggested “some type of waiver that students are able to sign 


for the purpose of sharing their information.” 


Local postsecondary administrators in two other states echoed such comments. “They ask 
for pass rates on certification exams,” one dean stated. “A lot of students take those [exams] 
after they graduate, so we may not be able to get those data or tie them to specific names. 
We could try to collect that afterward, but it would mean a staff person we don’t have [now] 
calling the student.” A counterpart in another state noted that documentation of skill attain- 
ment is more difficult when there is no national exam: “We rely pretty much on the grades 


and whether or not [students are] passing through their programs.” 


At the state level, one CTE director pointed to “a lot of technical assistance with locals” on 
technical skill attainment. She also acknowledged that the use of nationally recognized exams 
is cost-prohibitive for some LEAs because they would have to pay for their students’ exams. 
A postsecondary administrator in another state said, “The problem with using third-party as- 
sessments is that it can lead to ossified curriculum.” He also cited data access challenges. 
“T’ve tried to get [scores on] third-party tests [through administrative data matches] and [test 


administrators] won’t do it; nobody does it.” 


Many local practitioners reported challenges associated with the collection of placement da- 
ta. One area CTE director noted that high schools in the area “rely on an exit survey, which 
is pretty much what I plan to do. It’s probably about 40 percent accurate, if that good.” She 
said that her center does “as good a job as anybody in trying to find students and what 


they’re doing,” but indicated that it is almost impossible to find them in the state data system 
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without a Social Security number. A counselor at an area CTE center acknowledged both the 
importance of placement data and the difficulties in obtaining them: “I understand why they 
want to know. ... But people in our area who aren’t employed above a certain financial level 


tend to be pretty transient.” 


Another challenge identified by local respondents was the lack of connection between Per- 
kins reporting requirements and district data needs. “One of the challenges we face is the 
demands for Perkins reporting are not always in balance with the other demands of the dis- 
trict,” a local administrator observed. “We can’t work it into the work flow because it creates 
peaks and valleys.” She questioned whether it might be possible to alter the reporting sched- 
ule or adjust the fiscal year to spread out “those ebbs and flows.” In another district in that 
state, the superintendent talked about disproportionate demands on teachers’ time: “It’s the 
teachers in the classroom [who] are doing all of this... and it takes away directly from... 
what they can do with students. I know we live in a time of accountability, but the paper- 
work is a killer—and that’s on everything, not just Perkins.” Other local CTE directors 
pointed to the imbalance between the amount of Perkins funding received and the level of ef- 


fort required to collect the data. 


Almost all local respondents identified challenges in tracking students from secondary into 
postsecondary education. As one IHE administrator noted: “High schools want to know 
how their kids [do] when they get here, and we want to know what happens when they leave 
us, but we’re still having a difficult time tracking them.” Another local postsecondary admin- 
istrator concurred: “It really is the transitional data that are the hardest to get. Everybody 
knows [they’re] necessary. It’s a challenge [to get them] at just the local inter-agency level, 


and it’s an incredible challenge statewide as well.” 


Other reporting challenges identified at the state level included the timing of data, the varia- 
tion in data systems at the local level, and data accuracy. One state-level administrator noted 
the absence of a common data system at the secondary level and suggested that there are 
about 20 different systems in use in the state. “There’s no electronic upload because every 
high school has a different data system,” he said. “We collect all of our secondary core indi- 
cators through a separate online data collection system and all the LEAs have to hand-enter 
all that data on enrollments.’ He admitted that the Department had asked how he knows 
that the data are valid and reliable. “Our answer was [they’re] about as valid and reliable as 
we can get, because [they’re] the only data we have.” A state administrator in another state 
reported similar concerns about the data collected at the local level: “The problem that the 
locals are having is in data reporting. .. . I think we still have some problems with accuracy, 


but I think it’s 100 percent better.” 
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Reporting on Special Populations 
While most state CTE administrators expressed confidence in their ability to report dis- 
ageregated data on most of the special populations identified in Perkins IV legislation (i.e., 
individuals with disabilities; individuals from economically disadvantaged families, including 
foster children; individuals preparing for nontraditional training and employment; single par- 
ents, including single pregnant women; and individuals with limited English proficiency), 
some raised questions about the accuracy and utility of these data. Some also indicated that 


their capacity to provide this information had improved between Perkins III and Perkins IV. 


Administrators in one state reported no difficulty in collecting data on race, ethnicity, Eng- 
lish-language learners, and special education status. The CTE director in this state acknowl- 
edged the importance of collecting data on the performance of subpopulations. “The intent 
would be to share them with the LEAs and analyze them at the state level to see if we notice 
any obvious gaps,” she explained. “For us, it’s really about are they performing?” However, 
the accountability director questioned the reliability of subpopulations data that were self- 
reported. “Anything related to teen parents, single parents, and economically disadvantaged 
can be difficult,” she acknowledged. 


Both secondary and postsecondary state CTE administrators in a state that had recently un- 
veiled a new data system expressed high levels of confidence in their capacity to report some 
subpopulations data in greater detail than asked for in federal reporting requirements. Not- 
ing that the state had “lots of capacity” for reporting disaggregated student populations, a 
secondary administrator cited the example of moving from eight racial categories to 64 racial 
subgroups. A postsecondary colleague reported that the state data system was designed to 
distinguish between multiracial subgroups. “We’re actually a little step ahead of the Integrat- 
ed Postsecondary Education Data System (IPEDS),” she suggested. “IPEDS has a category 
that takes everybody who’s multiracial and puts them in one bucket, but when we do our re- 
porting, we take it down to the different combinations. We get a finer grain that we can roll 
up to the IPEDS.” Although she also noted that reporting on some populations is based 
more on “deriving instead of taking the face value data,” she commented that these catego- 
ries can signal opportunities to improve the system. As an example, she noted that limited- 
English-proficient students typically are identified by flags on English-as-a-second-language 


courses. 


A third administrator in this state said that special populations “do well for everything that’s 
in the management information system” but conceded that some special populations cannot 
be identified through the system. He reported that state Perkins administrators “have trouble 
with” displaced homemakers and pregnant or parenting teens. Every year on the pregnant 


and parenting teens, we write in and say we don’t collect this.” 
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A postsecondary official in a third state indicated that “special populations data are usually 
collected through self-reporting. For the most part, I think they’re valid, but I think no one 
understands what a displaced homemaker is.” In yet another state, a postsecondary official 
commented on the reporting challenges posed by the “temporal” nature of single parent and 
displaced homemaker status: “Because you can be [in one of] those [categories] one semester 
and not another semester, [the IHEs] have to ask those [questions] every semester.” His col- 
league at the secondary level reported the capacity to disaggregate by ethnic groups and oth- 
et special populations to compare their performance with all students but acknowledged that 


“wwe don’t have the whole system refined to getting student-level data.” 


Administrators in another state were quick to point to improvements in their capacity to col- 
lect and report special populations data. The state CTE director said, “We’ve struggled in the 
past because we’ve not had a decent state data system to draw from.” Another administrator 
explained that the new system allows districts to submit a “student snapshot file” that in- 
cludes race, ethnicity, free and reduced-price lunch eligibility, and eligibility for services un- 
der the Indwiduals with Disabilities Education Act (IDEA): “What we’re able to do is generate 
how many agriculture participants we had and then break them up by their subpopulations 
and identify all [those] kind of data.” 


Local Capacity to Collect Special Populations Data 
Local CTE practitioners tended to be less sanguine about their capacity to provide infor- 
mation about special populations, more explicit in their descriptions of the challenges associ- 
ated with collecting it, and more candid about their reliance on sources that could not be 
verified. Although school district officials generally reported becoming more adept at collect- 
ing information on different special populations, many also identified areas for further im- 


provement. 


In one state, local CTE directors in three secondary districts reported the capacity to gener- 
ate most special populations data. “Our data are all broken out by disadvantaged, disabilities 
... it’s very simple to look and see,” one local director observed. “Each teacher goes in and 
fills out about five questions about each student.” His counterpart in a much larger district 
said, “We have someone reporting on all of [these populations], and I have a great deal of 
confidence in these data.” However, the CTE director in a third district suggested that it is 
harder to get reliable data on some of the populations: “I think our data [are] good based on 
the system. .. . Because it’s pre-populated now, you can find that information. . . . It’s listed. 
If they’re an English-language learner, we know that. .. . The only three areas that are left 
open to ask [about] are [foster care, pregnant teens, and single parents]. Foster care... those 
are hard data to get... unless you just ask the student. [They’re] not reported. If I’m a foster 


patent, it just lists me as a parent. Pregnant teen? ... And how do you ask that in a class?” 
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In another state, area CTE center directors raised concerns about the quality of special 
populations data, suggesting that they varied according to the center’s relationship with the 
student’s sending high school. Center directors and faculty expressed high confidence in the 
quality of data obtained from the comprehensive high schools with which they share a cam- 
pus, but they also acknowledged the limitations of the data available. One area CTE director 
indicated that instructors use individual education plans, free and reduced-price lunch eligi- 
bility status, and other data to identify members of special populations. “It’s very rare we 
miss anything, especially within our host school,” an instructor commented. “Sometimes 
with our sending schools, we might not find out.” He explained that he made assumptions 
“based on what we’re seeing and the feedback we’re getting from the kid” to identify disad- 
vantaged status if the student was not enrolled in the free and reduced-price lunch program. 
He also said that he generally found out about single parent status “within the first six weeks 
or so, especially when it gets into scheduling, babysitting, and things like that.” A guidance 
counselor at another center indicated that she usually was able to draw information on sub- 
populations from the school’s data system: “I’m actually very fortunate because I know how 
to mine the system. I’m pretty tech-savvy, and I’ve worked with the system for several 


years.” 


The director of another center in this state expressed more explicit concerns about the quali- 
ty of the special populations data. “We do our best to gather subpopulation data based on 

kids’ reporting,” he stated. “We sometimes don’t know who fits in which group. It’s hard to 
gather subpopulations data because you don’t know. For example, we don’t if somebody is a 
single parent until the bitter end. It’s very possible that kind of data skips by us.” He also in- 
dicated that he has a “better handle on the number of students with disabilities [because] we 


have meetings to discuss individual education plans with sending schools.” 


Similarly, local postsecondary administrators reported that their capacity to provide accurate 
data varied across Perkins subpopulations. Several confirmed challenges identified at the sec- 
ondary level and indicated that self-reporting undermined data quality. For example, one lo- 
cal postsecondary administrator stated that her institution did a pretty good job of collecting 
information on disabilities, gender, race/ethnicity, and single parents but had a much more 
difficult time identifying displaced homemakers. “We can only collect data on single parents 
from financial aid applications, which [cover] only about 75 percent of single parents,” she 
reported. “Disabilities and ethnicity are self-reported, [but] most of our students do fill out 
that information.” Another IHE administrator in this state said that he used similar data 
sources for Perkins reporting. “We are reporting only single parents who have applied for fi- 
nancial aid,” he said. “Economically disadvantaged is anybody who’s spending full Pell [fi- 
nancial aid grants]... . Students with disabilities self-report.” He also noted that his 


institution did not collect data on displaced homemakers. 


APPENDIX C. CROSS CASE REPORT C-71 


A local postsecondary administrator in another state indicated that she had a high level of 
confidence in data collected and tracked on campus but distrusted data obtained from exter- 
nal sources. Noting that some data tracked on campus stem from self-reporting of disabili- 
ties or self-identification as a foster youth or a displaced homemaker, she asserted that “any 
student who self-identifies in a category is easy to track.” However, she also acknowledged 
that self-reporting can often be equated with underreporting. “There are all kinds of studies 
and statistics about the theoretical number of students with unreported learning disabilities 
ona college campus,” she explained. “They may not self-identify because they don’t want to 


share; they may not self-identify because they don’t know.” 


The Burden of Perkins Accountability Reporting 
Several administrators at both state and local levels were quick to note that the cost of Per- 
kins accountability systems far exceeds the human and financial resources available for this 
purpose and thus poses a substantial burden. Some went as far as to suggest that the ac- 
countability requirements represent an unfunded mandate. One state CTE director and local 
administrators in at least two other states reported that some subgrantees were giving up 
Perkins fands because of the burden and complexity of reporting requirements. A local post- 
secondary administrator said that one IHE that used to participate in Perkins had decided 
that the reporting requirements were too burdensome for the amount of money received. 
“They threw in the towel,” he observed. “They weren’t getting very much money .. . about 
$150,000 a year, maybe $180,000, in that neighborhood. They just decided it wasn’t worth it 


anymore.” 


A CTE director in another state reported similar observations about eligible recipients opt- 
ing out of Perkins funding: “If they want this mountain of data and they want technical as- 
sessments, they need to pay for [them]. I know some people opted out of even applying for 
Perkins; they didn’t think it was worth it.” A state-level postsecondary administrator in this 
state reinforced his point: “How would you feel if someone gave you less than two percent 
of your funds for operating your business and walked in and said ‘’m going to tell you how 
to run it?”’ He added, “There are lots of them at the program level who don’t want the 


money because they don’t believe . . . that it’s worth filling out all the paperwork.” 


The state’s secondary Perkins coordinator concurred, noting that “we have actually had local 
districts .. . release money because [they say] ‘it’s too much work for the money you’re giv- 
ing me.”’ He suggested that the burden of Perkins reporting requirements was a major impe- 
tus for reorganizing state staffing for CTE. “At the state level, a lot of the reason for creating 
the new Perkins unit and increased staffing had to do with the burden of all the new require- 
ments, not only under Perkins IV but even under Perkins II,” he reported. “It used to be we 
collected data, and we sent it to the feds aggregated for the state. That was the end of it. 
Now we not only have to aggregate all those statewide data... but I’ve got to look at each 


separate [local] agency and identify those that are low-performing.” He asserted that the vol- 
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ume of work exceeded the capacity of the two people then assigned to monitor all local pro- 
grams, which led to the decision to create a Perkins unit within the state department of edu- 
cation. Pointing to the reporting requirements of ESEA, he noted that it is easier to “force 
districts through large dollars to abide by those accountability measures” but asked, “How 
can you push that on districts for [a much smaller] dollar amount?” He said that Perkins data 


collection has “been very burdensome, especially for the locals.” 


Secondary CTE officials in two other states shared his assessment. “I think they’re feeling 
the burden at some of the small schools,” one state CTE director observed. “At one of our 
tiny area CTE centers . . . the guidance coordinator is the data person and also serves as the 
equity representative, and I think she’s also the safety coordinator or something, . . . So it is a 
burden, particularly for that center. They only get about $38,000. ... That doesn’t even pay 
her salary.” He also noted that at least three of the area CTE centers in the state had men- 
tioned the possibility of dropping Perkins funds. “They’re saying this just isn’t enough money 


to warrant this upheaval.” 


The director of professional development for the eligible agency in another state suggested 
that the Perkins burden was not limited to small, rural districts. “I think the biggest burden 
[the LEAs] have is keeping up with the data,” he stated. “It’s really difficult; I think the prob- 
lem is the method or system that’s used and the communication to the key people on how to 
go in and gather data.” He reported that teachers typically provide information through local 
data administrators who can make their own decisions about how to code certain things and 


may have different perceptions about what is important. 


A state administrator in another state suggested that the burdens of collecting data differed 
at the secondary and postsecondary levels. He acknowledged the effort required for local 
districts to hand-code completers, obtain Social Security numbers, enter technical skills as- 
sessments, and submit a completer file for the follow-up survey conducted by a third-party 
vendor. He acknowledged that we have to struggle with “every one of those things. . and we 
have a general problem. In contrast to [postsecondary], secondary is understaffed for data 
quality control and even understaffed for analysis.” A postsecondary colleague at the state 
level, however, reported needing additional staff to comply with Perkins reporting require- 
ments: “We used to have one staff member [who] did it, and now we have two because there 


are so many extra pieces.” 


An official in another state suggested that the state’s budget crisis and subsequent staff cuts 
had made Perkins data collection and reporting more difficult. Even so, she regarded the da- 
ta-reporting requirements as important, noting that it was unrealistic that the state should 
collect millions of dollars in federal funds without having “to be accountable for [them].” 
After speculating that reporting requirements were probably “less burdensome on us than 


some states,” she acknowledged that “we would have to do that for the state money that we 
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get, regardless of whether or not we get any federal money. Everybody has a management 
information system in this state; [Perkins] doesn’t force them to have a system. But at the 


margins, it can produce work for everyone.” 


In contrast, the CTE director in another state said that data collection had become less bur- 
densome over the past five years. He attributed the change to a new state data system, noting 
that data were previously “done by hand .. . at the school level, with people going through 
transcripts and counting courses. Once we got the statewide longitudinal data system in 
place, suddenly data became less burdensome and more valuable, because you can pull 
[them] out of the data warehouse, you can manipulate and get down to what you really want 
to know.” A colleague suggested that local districts probably would have said the new system 
was burdensome in its first year, “but now I think they kind of reached this point of, ‘Oh, 
this is just telling you who our kids are’ and then we give them all the data back. .. . So when 
we provided that tool and resource to them for free, that was kind of a big win perceptually 


for the districts.” 


At the local level, district and IHE administrators often complained about the burden im- 
posed by Perkins reporting requirements, and many questioned the effort required to pro- 
duce data that were generally not perceived to be useful. This pattern generally held across all 
of the case study states, regardless of the level of sophistication of the state’s Perkins report- 


ing system. 


In a state in which the data system was widely touted, one secondary local staff person with 
data-reporting responsibilities said, “It’s time-consuming the way the system is now—the 
process, the fields we have to enter. Some of them are redundant; you’ve answered some- 
thing in one, you’re repeating it in another, and they kind of have to match.” She suggested, 
however, that she anticipates the process becoming easier in the future. “With this new sys- 
tem or new data, I can see light at the end of the tunnel,” she remarked. “I’m hoping that it 
will be easier, and my intent is accuracy. But I can only be as accurate as the information ’m 


given, so there are ways to cross-reference what the teachers submit.” 


Several local CTE administrators commented on how Perkins accountability requirements af- 
fected district data or information technology staff. “District-level people have been very 
good when I’ve asked them for these multiple reports, because there are some things we 
can’t [do] here in our office,” one local CTE director reported. “They might roll their eyes 
when they see me coming [and say], ‘Oh now what does he want?”’ A local Perkins coordina- 
tor in another state said that reporting is not hard, just time consuming and indicated that 


the LEA’s “information technology department is always overwhelmed.” 


Other local CTE administrators voiced frustration about the amount of time they devoted to 


reporting responsibilities. One local CTE director noted that much of his job “is managing 
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data.” While acknowledging the need for data that show the “funding is working,” he also 
noted, “The more time we spend on that, the less time we have on actually teaching kids.” 
The principal in that district said he wished that federal and state officials could streamline 
their reporting requirements. “Different indicators are worded a bit [differently], although 
they might want the same information,” he noted. He also suggested that budget constraints 


exacerbated the difficulty associated with data collection requirements. 


Local staff and administrators in another state expressed similar concerns about the time de- 
voted to collecting and reporting data. One counselor described the state’s reporting system 
as “very antiquated” and noted that “it takes hours to input a lot of information.” After hav- 
ing worked with a temp for days to enter information at the start of the school year, she 
commented, “The fact that we don’t have a web-based system in this day and age baffles 
me.” She also noted that the absence of live files makes it cumbersome to change infor- 
mation: “If I have to make a change or if I’ve made a mistake, they have to call me and tell 
me to change mine, and they have to change theirs at the same time to make sure that both 
of them stay consistent.” Although teachers at this area CTE center played a major role in 
collecting placement data, the CTE director noted that a small percentage of CTE teachers 


had refused to conduct follow-up phone calls from home at night. 


The CTE director in another area center in this state found the reporting requirements less 
burdensome. “Data reporting responsibilities are more than reasonable,” he remarked, not- 
ing that he has an employee who collects data locally and submits them to the state. His 
counterpart in another community commented on the changes to the reporting requirements 
and level of effort associated with Perkins funds: “Back in the day, if we didn’t use the data, 
we didn’t report them. Some years ate worse than others in terms of the hoops we’re te- 
quired to jump. Next year ... it’s going to be probably the most severe in terms of tying the 
money to the accountability data.” He also said that his institution did not spend “an inordi- 
nate amount of time in order to draw the [Perkins] money down [from the state department 
of education. ... Right now the balance seems to work pretty well for me... . From my per- 
spective, $200,000 is $200,000, so... it would have [to have] a lot of strings attached to for 
me to say ‘I don’t think Pll take this.” 


Local administrators in another state were similarly divided about the burden associated with 
Perkins accountability requirements. “A lot of what we have to do for the state is paper- 
work,” one local CTE director noted. “Now I have money in there for these three assistants, 
one at each school, to help me with the reports.” He went on to emphasize the impact of 
Perkins reporting on teachers. “There is no question that it is time-consuming for teachers,” 
he asserted. “They have so many test scores to deal with already.” Another local administra- 
tor in this state said that the reporting requirements were “overall, not a huge burden,” but 
raised questions about quality control in data entry, particularly in districts with many high 


schools and a transient student population. 
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Local postsecondary officials echoed many of the themes articulated at the secondary level, 
with emphasis on the time and effort required to produce Perkins accountability data. Some 
acknowledged the need for documenting the results attained with Perkins funding, and at 


least a few indicated that the information produced is valuable. 


“T could be real glib and say all the reporting [is a downside of Perkins], but if you’re taking 
somebody’s money, they want to know it’s well spent,” a local IHE administrator said. “So 
reporting is reporting. .. . As a receiver, I would love to see less stringent reporting and 
more trust, but I understand why the constant improvement must be shown [and why insti- 
tutions] need to report in all the areas. ... But that’s what is time consuming on my part and 
means that I don’t get to do the actual work in the field.” A counterpart at another [HE in 
this state reported that she was able to coordinate Perkins reporting with another federally 
funded program that collects similar data: “So we’ve got some extra staff trying to collect 
that information, so it makes it a little easier for us. [We] don’t have a designated institutional 


research person, but [we] do have another staff person who fulfills some of that role.” 


Two local IHE administrators in another state reinforced the claim that collecting Perkins da- 
ta is labor intensive and therefore expensive. Explaining that the state system that aggregates 
student information is “very, very, very old,” one noted that the system posed staffing chal- 
lenges. “They were going to re-host it 15 years ago .. . and it’s never happened. It’s so cum- 
bersome for the colleges to get any data out of it.” A colleague in the same community was 
much more explicit in attributing additional costs to Perkins accountability requirements. 
“Both of our institutions have now hired a researcher to track data for us,” he reported. “It’s 
a vety, very big expense. This is again my bias .. . that the outcome is higher reporting as 


opposed to higher accountability.” 


Local IHE administrators in a state with an advanced data system also questioned the value 

of the measures used in the Perkins accountability system, and one suggested a graduated re- 

porting system tied to amount of funds received. “If I’m a legislator, I want to see where the 
money I gave you had an impact... and what that impact was, instead of just some arbitrary 
measurement,” remarked a dean at one IHE. Another IHE administrator proposed an alter- 
native funding strategy to accommodate reporting costs: “An alternative would be to distrib- 
ute the money [in a way] that those who received a certain dollar amount or less weren’t held 


accountable to all these standards that are very challenging.” 


Using Perkins Accountability Data 


Although some state and local respondents reported using Perkins IV data for program im- 
provement, others indicated that they made little or no use of these data. Very few were able 


to identify uses of special populations data. 
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States 


Several state CTE administrators could identify specific ways in which they used Perkins ac- 
countability data to identify areas for improvement. One secondary state CTE administrator 
referred to accountability data as the “statewide report card” for secondary schools. She said 
that examining the data enables staff to map out deficiencies and work to boost enrollment 
and scores. The CTE director in another state described initial efforts to disaggregate Perkins 
performance indicator data to the level of CTE participants or concentrators and then to 
some of the special populations (e.g., race and ethnicity). “When we do that, there are some 
horrible performances coming out,” he reported. Citing a longer-range plan to provide dis- 
ageregated data on academic performance to all secondary schools in the state, he said his 
preliminary research identified 14 or 15 high schools in which “100 percent of the students 
that they sent to area CTE centers were at the lowest proficiency level in both math and 


reading.” 


In a third state, a secondary state CTE administrator described use of instructor-level data to 
improve teacher performance: “We have been able to look at programs and see if we 
thought the teachers were doing a good job and make recommendations for professional 
development in a much more prescriptive way than we’ve been able to in many years.” Offi- 
cials in this state also noted innovative plans to help local subgrantees make better use of 
Perkins IV data for program improvement. A member of the state’s Perkins leadership team 
indicated that state staff members are developing a data tool for producing customized per- 
formance reports for local subgrantees: “One of the things that we hope to launch for Per- 
kins is a business improvement project. ... We have the [reports] by school, by program, and 
then the sub-indicators that we hope to be able to share to give [subgrantees] the resources 


they need to identify problems.” 


Administrators in one state reported changing its local Perkins application to encourage dis- 
tricts to reflect more on accountability data and use them to make funding decisions. These 
officials also said that they emphasize the importance of using data when they provide 
statewide technical assistance to districts. The CTE director noted, “We’re constantly taking 
[district staff] back to the data, and that’s just paramount.” His deputy provided a more con- 
crete example of how an analysis of special populations data led to a decision to target non- 


traditional expenditures at the middle school level. 
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Using Perkins Data for Programmatic Change 


A state-level administrator described using Perkins data on nontraditional training and employment to 
target funding more effectively: 


[W]e have flat-lined on our nontraditional forever. No matter what we did, about 14 percent of our 
participants were nontrad by gender. . . . We put grants out; we did all kinds of stuff. . .. And | don't 
think we were wasting the money, but we weren't doing anything significant enough. So a year anda 
half ago—we’d always put our nontrad money out in competitive grants—we took a year off and did a 
major study that said, what are the absolute barriers that are out there? What's an analysis of how 
we’re serving special populations? We hired a consultant. She did a year’s worth of work. She did 
focus groups; she visited all the places. One of the things that we discovered was we were spending 
our money at the back end of the system, a lot of community colleges, etc. The research says, "If 
you’re going to change attitudes about gender stereotypes, start on the front end. This is a Band- 
Aid.” So we've shifted [the way] we’re doing our funding [now] to say, “We’re going to do more work 
with middle school. We’re going to do more work with our career information system.” We want to 
make sure that there is nontraditional information on that system that encourages students to use 
that to explore careers. 


Another state secondary CTE administrator commented on the need to “get out to the Per- 
kins cootdinators and have them look at their own local data” and observed that academic 
teachers were introduced earlier to the importance of data because of the NCLB legislation. 
“Academic teachers and administrators are really starting to use the data for improvement, 
but I think, with CTE teachers, we’re just starting to do that,” he explained. “CTE teachers . 
.. have often times not been trained to use data because under NCLB there were no CTE 
data. Some of the CTE teachers haven’t even been exposed to using data to improve instruc- 
tion.” This administrator also said that Perkins data are shared annually with the state legisla- 


ture. 


A postsecondary colleague emphasized the critical importance of collecting data about single 
parents because they have special barriers. “We can do things about those barriers,” he said. 
“We provided evidence that if you collect this information, you can make big differences.” 
He noted that some colleges that have been collecting data on single parents for a long time 
have implemented programs to break down the barriers: “So their students in those catego- 
ries are doing fine, and they watch [that they don’t] fall below the mainstream performance 


levels.” 


In contrast, state postsecondary administrators in two states suggested that Perkins data were 
only used to secure continued funding. As one summarized, “I think that the attitude among 
states is... if [the Department] wants some numbers, we'll give them numbers—not that 
they’re necessarily valid or reliable.” His sentiments were echoed by a postsecondary coun- 
terpart in another state. “Do we use them for policy?”, she asked rhetorically. “No. We’re 


collecting the data for compliance reasons.” 
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LEAs 


While some district staff discussed analyzing and reflecting on the Perkins accountability data 
to improve their programs, the majority reported not using the data. Several local administra- 
tors in one state, which had invested significantly in a new database, explained that Perkins 
data only reveal what they already know. As one local superintendent observed, “Somebody 
in Washington [may know] that we have 17 kids. .. . But that doesn’t do me a darn bit of 
good. ... It’s not helping us accomplish our mission in helping educate kids.” The high 
school curriculum director reinforced concerns about the value of Perkins data compared to 
the effort required: “For the effort put in for what we’re getting out of it, it doesn’t seem like 
much, I mean it’s like working all day for a dollar.” However, a CTE director in another dis- 
trict said that both administrators and instructors use Perkins data to reflect on their pro- 


grams. 


In contrast, a local CTE director in another state reported extensive use of Perkins data. He 
said, “We brought all the principals together and shared the data with them so they could see 
the progress that their CTE students have achieved.” Another local CTE director in this 
state linked data to schools’ ability to help struggling students: “In our types of programs .. . 
we get kids who are right on the cusp. .. . It’s the data that help us know how to help those 
kids and push those kids in the direction they need to go.” 


Three local CTE directors in a third state explained why they did not use Perkins data at the 
local level. One conceded a compliance approach to Perkins data: “I do what’s asked for, get 
[the data], look at them, and report them rather than act on them.” While she said she paid 
attention to numbers that dropped between semesters and looked at information related to 
special populations, she observed that Perkins data were not “always aggregated or disaggre- 
gated in a grouping that will meet my question.” Another indicated that there was no reason 
to use the data because the district met or exceeded its performance targets. He indicated 
that the data might be shared with teachers in the future if the district’s performance drops: 
“Tf it should come to pass in the next application that we have to address a certain area, then 
... Twill photocopy [the data] and see if any of my teachers want to look at them.” The third 
questioned the state’s “shotgun way of approaching” data collection: “The feds told us we 
have to report something, so here are the numbers. . . . If the goal is really to improve our 
programs, I’m all for that, but I need accurate information in order to do that.” Nonetheless, 
this official also cited local plans to create a “data dashboard” system that would allow 
teachers to look at how many special education or free or reduced-price lunch students they 


have in their classes. 


Local CTE directors in another state identified similar reasons for limited use of Perkins data. 
“T haven’t had to use [the data] for program improvement,” one local CTE director asserted. 
“T look at the data and share them with teachers annually so we know where we [are], but we 


pretty much already know it anyway, because we try to pay attention to what’s going on in 
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IHEs 


our programs.” Another local CTE director said that she didn’t use data for program im- 
provement, indicating that they were more useful to state administrators for monitoring pur- 
poses: “I think . . . it’s making sure that that we’re following the regulations that the state sets 
and that we’re utilizing the money correctly.” A third local director suggested that the data 
were linked to future funding: “Mainly, they’re used for us to earn Perkins dollars for next 
year... to show that our programs are successful enough with students that we can continue 


to receive funding.” 


A local CTE administrator in yet another state was more explicit about why he did not use 
disageregated Perkins data. “I have no confidence in that report because I don’t know how 
it’s put together,” he stated. “I have to gauge our work with subpopulations by talking with 
teachers, but I can’t report on it... . The composite data are a mess; we don’t have good 
composite data.” He went on to describe disaggregation of data in the state as “garbage in, 
garbage out,” noting that there is “lots of distrust” about data. He said, “I probably couldn’t 
tell you how many special education kids have dual enrollment. There is no single place that 
data are collected together and can be manipulated. Data are in separate places. It all de- 


pends on the question you want to ask.” 


Perspectives on the value and utility of Perkins data also varied at the postsecondary level. A 
few IHE administrators provided examples of how they used Perkins data to identify oppor- 
tunities for program improvement, but more respondents indicated limited use of these data 
and questioned their value. Several others said that Perkins data were a starting point from 


which they conducted more in-depth local analyses and reported that local data were more 
useful. 


In a state with a more sophisticated data system, one IHE administrator displayed a publica- 
tion that links Perk7ns data with institutional data to provide summaries of outcomes, includ- 
ing graduate earnings, by program area. “We’re getting most of the Perkins accountability 

data at an aggregate level,” he reported. “When we’re trying to disaggregate it, we’re doing it 
from our local data.” Noting the absence of a direct match between the two data sources, he 
described the process of taking Perkins reports generated at the state level to “maneuver and 
manipulate ... into the measures that we’ve established for program effectiveness at the col- 


lege.” 


A counterpart at another IHE in this state said that “locally generated evaluation data are 
more useful than reports from the state. We’re more interested . . . at our local level [in] what 
our students ate saying... . State data... tell us where we’re retaining well and that sort of 
stuff... but we already know that. Why are we not retaining well in some programs? I want 


to talk to the students who are leaving. I don’t want aggregated information from the state.” 
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Local IHE administrators in another state agreed that locally generated data were used more 
frequently and raised serious questions about the quality of data generated at the state level. 
“Tt’s really hard for us at this institution to talk about Perkins data because we’ve tracked all 
of this stuff just for ourselves,” one IHE official commented. “We have a program effec- 
tiveness process that looks at each of our programs, and each faculty member works with 
that.... We can disaggregate that more on our campus to look at particular populations.” 
He also noted his cynicism about the state data system and his distrust of the information it 
produces: “TI think that the accountability stuff has created a whole industry of researchers. . . 
. I don’t think it’s doing great economic development things for our communities.” He said 
the “state provides this institution with lots of data, and then we have to find some addition- 
al data for the report. .. . When I went through the report, I thought ‘We’ve got some data 
problems; we’ve got some coding problems.” Nonetheless, he acknowledged that Perkins 
data were used to redirect resources when the college was notified that it met enrollment cri- 
teria but not its completion targets: “It did allow us to look at that specifically, to put some 
more resources into that to change those numbers.” A colleague in this community indicated 
that she used Perkins data for program improvement and to direct resources, whether it’s to 


close a program or to fund something else. 


In another state, a local postsecondary administrator pointed specifically to the limited use of 
special populations data in his institution. “Internally, we’re not too interested in these num- 
bers,” he acknowledged. “Program managers are interested in the total number of concentra- 
tors and participants in their programs, but they aren’t that interested in these special 
population numbers because a lot of our students fall into one of these categories.” He ex- 
plained, “We don’t see significant differences in outcomes for subpopulations. For example, 
completion was 53 percent overall. For individuals with disabilities it was 50 percent... sin- 


gle parents 54 percent, limited-English proficiency 58 percent.” 


In at least two states, IHE administrators expressed concerns about the accuracy of Perkins 
placement data, particularly for employment. “Some of those data are really not all that accu- 
rate and useful,” one [HE administrator remarked. “They’re better than nothing, but they 
leave a lot out. They lag in time. They leave out anybody self-employed . . . and for some ar- 
eas like IT, they’re completely useless. You can’t tell if someone’s working in IT; they may 
report that they’re working in healthcare, but they’re in health IT.” A peer at an THE in an- 
other state concurred, “[W]e’re still looking at our placement measures, recognizing that 
theyre the best data available, but they’re not the most accurate. They’re two years old... 
and they’re [limited] to those that were found.” Both of these respondents cited the limita- 
tions of placement data derived from administrative records matching because they cannot 
identify whether a former student is employed in a training-related occupation. As one ob- 
setved, “Some of the larger employers .. . have divisions that would cross many, many dif- 


ferent occupations and many different training programs.” 
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Adjusted Performance Levels 


As was the case with Perkins II, Perkins IV specifies adjusted levels of performance as the 
mechanisms for tracking states’ performance on each of the core accountability indicators. 
Statutory language requires eligible agencies to identify “objective, quantifiable, and measur- 
able” (Sec. 113(b)(3)(A)) levels of performance expressed in a percentage or numerical form 
for each of the core indicators as well as plans for improving the performance of CTE stu- 
dents, as measured by these indicators. Perkins IV does not define “adjusted levels of per- 
formance,” but interviews with state CTE directors suggested that performance levels vary 
across states and target continuous improvement in successive years. The Department, act- 
ing on behalf of the secretary of education, negotiates performance levels with eligible agen- 


cies as a condition for receipt of Perkins funds. 


As noted previously, Perkins IV extends the requirement for negotiated performance levels 
to eligible recipients. By requiring state CTE officials to negotiate performance levels with 
LEAs and IHEs, Perkins IV creates a two-tiered accountability structure. State administrators 
must negotiate sequentially more challenging targets for state performance on the core indi- 
cators, but they also are given a new tool for holding eligible recipients accountable for their 


respective roles in meeting the state’s performance levels. 


State Negotiations with the Department 
State administrators reported a wide range of experiences in negotiating performance levels 
with the Department. Administrators in most states expressed some frustration with the 


process and suggested ways to improve it while others commended the Department. 


One state assessment director described the negotiation process as “proposal-counter pro- 
posal conducted via email with the regional accountability specialist [RAS]... . There was 
apparently no room for negotiation on 181, 182, and 481 with our RAS. It was quite clear 
that we were going to be held accountable for the [ESA] standards that had not been ne- 
gotiated for CTE ... and we did capitulate.” The CTE director in another state explained 
how frustrations with the negotiation process almost led his state to pull out of Perkins par- 


ticipation: 


[Our] population is not your typical secondary school population; we’re 
oversubscribed in special education. So [our RAS] got that [we] couldn’t 
just follow [ESA] growth charts, yet still negotiated fairly. For instance, 
for math in the CTE population, we were at 11 percent proficiency. ... We 
had negotiated 20 or 22 percent proficiency, even though we knew we were 
only at 11 percent; then they came back and wanted 35 or 37 percent. ... 
Neither our agency nor [the Department] would budge. I said, “Then I 


don’t know what options we have here, except we’re not going to take the 
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money if it’s based on this. This is crazy. I can’t agree to this.” ... It was 


not a rational discussion at all. 


This administrator explained that the state agency ultimately ended up setting a 22 percent 
proficiency target (for secondary academic attainment indicators) and expressed further frus- 
tration about what he perceived to be the Department’s reluctance to collaborate on longer- 
range solutions to this reporting challenge: “I offered to fund a statistical review to map out 
a five-year plan of what would be realistic for our cohort of CTE students, but [the Depart- 


ment] wasn’t interested in that approach.” 


Another state CTE director had similar perceptions about the Department’s flexibility in the 
negotiation process: “It felt like there was this random push to elevate everything. We want 

to be challenged by the [performance levels], but don’t make them unrealistic. In our negoti- 
ation, I think the comment was made, “This is what is expected, so I really can’t even negoti- 


ate with you.” 


The Perkins coordinator in a fourth state gave the Department good marks for the negotia- 
tion process but raised broader concerns about the content of the negotiations. He described 
the negotiations with the Department as very professional: “We made some suggestions, ex- 
plained why we made those suggestions, and [the Department] responded.” He also noted, 
“T don’t like that the accountability measures are target-based. .. . Maybe Congress needs to 


say ‘we would like to know this’ rather than saying ‘collect this information.” 


A state assessment director also wanted to see better explanations from the Department 
about what information is needed and how targets are established: “It’s not really clear what 
methodology they’re [using to select] a target. If it was more transparent [and]... we knew 
what their goal was, we... could figure out what to do. ... It also makes if difficult when 
there’s supposed to be coordination with a liaison ... when we don’t know by what standard 


the negotiations are occurring.” 


This individual was one of several state CTE administrators who emphasized that 100 per- 
cent targets were unrealistic: “The negotiations .. . were based on reaching 100 percent... .. 
We all know that was not a reasonable supposition that anyone will ever reach 100 percent in 
[the] eight-year period that was negotiated under [ESA], but we were held to those [tar- 
gets].” In another state, a postsecondary administrator with assessment responsibilities was 
even more direct in his criticisms of the 100 percent targets: “I think it’s irrational to say 100 


percent of all students entering your institution will be in each measure.” 


A postsecondary administrator in a third state concurred: “The feds think 100 percent on 
everything is reasonable in a 10-year growth model and it’s not.” He was somewhat more 


positive, however, in his assessment of the negotiation process with his state’s RAS: “We 
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didn’t get our way the first time, but we talked about what’s reasonable. What could we look 


at as far as a growth model? I’m very pleased; working with him has been good.” 


State Negotiations with Local Subgrantees 
Perkins IV requires eligible recipients to accept their state’s adjusted levels of performance or 
negotiate their own for the appropriate set of core indicators. Although Perkins IV suggests 
that the state process for negotiating adjusted performance levels with eligible recipients mir- 
rors the one used by the Department for negotiations with states, significant variations in in- 


terpretation and processes were found across states. 


Administrators in three states reported that their processes for establishing local perfor- 
mance levels were not based on negotiations. A secondary CTE administrator in one state 
acknowledged, “We have defaulted to not really negotiating, but it’s more requiring that they 
meet a state goal.” A postsecondary state administrator in another state termed the process a 
“dictated negotiation” because locals were given the choice of adopting the state perfor- 
mance levels or adding 3 percent to their current performance levels. She explained that the 
3 percent option recognized that all participating postsecondary institutions weren’t at the 


state level but incorporated an expectation of improvement. 


The secondary Perkins coordinator in a third state questioned the value of negotiating local 
performance levels: “Negotiating with locals is ridiculous. Frankly, we didn’t do it. We told 
out locals you have to accept the state levels or ask to negotiate because that’s what the law 
says.” He noted two specific problems associated with local negotiations: the lack of state 
staff to perform the task and the risk associated with lowering targets for some districts. Of 
the latter, he said, “For every agency I negotiate a lower number, that puts the state target in 
jeopardy. If you truly looked at the core indicators of quality programs, why would I hold 
somebody else to a different standard?” This administrator reported that no school district 


had asked to negotiate local performance levels. 


A postsecondary Perkins administrator in the same state described a different approach to es- 
tablishing performance levels for postsecondary institutions. He said that his agency “adopt- 
ed [the Department’s] method for negotiating with our locals” but identified what he 
perceived to be one major difference: “We were reasonable. [The Department] was not. And 
in most cases, people met their targets.” He reported that only six of the 72 participating 
postsecondary districts failed to meet at least 90 percent of one or more performance targets 
this year and attributed this success to workshops conducted across the state on “naive fore- 
casting.” As an example of the challenges associated with setting realistic targets, he asked, 
“How do you... negotiate achievable targets that demonstrate continuous improvement in 
an economy that’s going to drop six percent [of its workforce] in the next year?’ Nonethe- 
less, he said, “We want the targets. We want the leverage with faculty of saying ‘I need con- 


tinuous improvement here, folks.’ This is not business as usual.” 
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CTE administrators in one state indicated that they “tried to mirror the process that [the 
Department] handled with us” and improve it. They reported developing explicit criteria for 
negotiation that were shared in advance with eligible recipients. The secondary state assess- 
ment coordinator noted, “We provided much more information than [the Department]— 
explaining the process and giving [the local subgrantees] scenarios and examples of the pro- 
cess.” The state CTE director concurred, “What I like about [this process] is that [locals] 
know the rules up front. So we don’t have the issue that we have with [the Department]. Are 


they negotiating with me differently? We are very consistent.” 


She also explained that a work group of local practitioners helped state CTE administrators 
design a primarily web-based process for establishing local performance levels. The state 
CTE assessment coordinator reported, “We developed an online system that locals can ac- 
cess to see their performance measures. To accept, they just click. To negotiate, they... 
submit an official request to negotiate, with supporting justification on that particular meas- 
ure.” At the state level, CTE staff reviewed requests to determine whether the state could af- 
ford to make adjustments. This analysis was said to entail running calculations and providing 
local administrators with recommendations, which were subsequently incorporated into 


formal notifications of adjusted measures once they were accepted at the local level. 


While this state’s negotiation process appeared to be the most sophisticated among the six 
states, other states used elements of its approach. For example, the secondary CTE staff 
member responsible for assessment in another state reported that teachers and local CTE di- 
rectors can access the state’s web-based data management system to view their past perfor- 
mance levels and decide whether to accept the state performance level or negotiate a local 


one. 


The CTE director in another state reported using a 10-year growth model to negotiate per- 
formance levels on an individual basis on all but the nontraditional indicators. He said that 
state staff encountered no problems because “we took a realistic perspective.” He confirmed 
observations from staff in other states about the importance of realistic goals, noting that the 
suggested local performance levels were based on current performance levels for both area 
CTE centers and IHEs. A postsecondary administrator in this state reinforced the im- 
portance of using past performance data to negotiate performance levels: “Our state target 
for placement is a lot lower than it used to be... because we knew when we negotiated 
these targets that having to count the leavers was going to depress these. We ran all the data 
from 2006-07 . .. on a trial basis to get a sense of what our performance measures were go- 


ing to look like and then we negotiated.” 


Local Perceptions of the Negotiation Process 
Reflecting the variation in state-level approaches to setting local performance levels, local 


staff had varying perspectives on the performance levels and process by which they were es- 
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tablished. School district and postsecondary administrators were generally aware of the ex- 
tent to which negotiation was an option in their states, but some were more concerned than 
others about what they perceived to be limited opportunities for input into the local perfor- 
mance targets. Several respondents cited the utility of using prior performance data to nego- 
tiate Perkins IV performance targets. Even within states, there were divergent perceptions of 


the process. 


Differences in Local Perspectives on One State’s Web- 
Based Negotiation Process for Establishing Local 
Performance Levels 


LEA and IHE administrators reported limited opportunities to negotiate local performance lev- 
els: 


Now the state sets our targets for us—and they raise those targets every year based on the 
performance we had the year before. We haven't negotiated; we’re allowed to, but we haven't. 
(Secondary CTE director) 


We've always exceeded what the state requires for minimum figures. When they require a 
certain percentage of placement, they do not take into consideration the unemployment rate— 
and our unemployment rate is hovering around 14 percent. We have a minimum of 70 percent 
placement in their fields, which is extraordinary with that type of unemployment rate, but they 
don’t take that into consideration. (Secondary CTE instructor) 


They came to us and said: “Here are your performance measures. Here’s what we’re looking 
at.” You either accepted the percentages that they said you’re going to meet, or you negotiat- 
ed with the state. If you negotiated with the state, it delayed the Perkins funding for your dis- 
trict. (Secondary CTE director) 


The state says we can negotiate, but we can’t. (Local postsecondary administrator) 


There were only two colleges in the state that negotiated. . . . We got the pretty clear message 
that there wasn’t much room [for negotiation]. (Local postsecondary administrator) 


We have outperformed the state in 11 of 12 of those measures. Therefore, we are in the 
group that are required to make nominal progress of .5 percent on our actual performance. . . 
. SO we are really at a point where we are competing against our previous year’s perfor- 
mance. .. . | think that the .5 . . . quantifies what's sufficient progress. . . . We didn’t try to ne- 
gotiate the targets. . . . We just accepted the local targets based upon improving on our 
previous year performance. (Local postsecondary administrator) 


We have a window of time when the application is released to go online and submit our appli- 
cation to negotiate, and there’s a form. We have a complete process. I’ve never done it. I’ve 
always just accepted the state indicators. | honestly do not have the time. . . . | looked at our 
past numbers when | made this decision and what the new numbers were and said it’s close 
enough. (Local postsecondary administrator) 


A local secondary administrator in one state described a constructive process that compels 


local CTE directors to focus on the current status of their programs: “The negotiations are 
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good because [they] cause some of the old directors to be more accountable . . . and make 
them better understand the process. When they have to sit down at the table, and they have 
to be able to talk intelligently about it... . I think we’re in a time where that is going to be .. 
. the big white horse in the room that everybody’s going to have to deal with... . I like the 


way the state goes about it: let’s negotiate so you can meet it.” 


A regional CTE director in another state reported similar satisfaction with the negotiation 
process: “We were given a sense of where we stood historically on those indicators ... and 
they said, ‘Make some projections for improvement over a three-year period.’ .. . Assuming 
the historical data were a good place to start, we added miniscule increases to show im- 
provement. ... The advice we got from the state was don’t make outrageous projections that 
you can’t possibly meet because you'll be writing improvement plans in the future.” Another 
director in that state said that he wasn’t forced to negotiate because his center was above 
state averages: “We set goals slightly higher but not too much. So far they have had no im- 


pact on our school.” 


Secondary CTE directors in another state saw no options for negotiating performance levels 
with state officials. A CTE director in a small rural community said, “From our little school 
to the state, there’s no negotiation. ... Here’s our number, here’s the state number. We have 
to be within 80 percent of something, [but] we’re blowing the state numbers away anyway.” 
The Perkins coordinator in another LEA was even mote direct: “It’s either you meet it or you 


don’t.” 


Local practitioners in two states argued for greater transparency in how state administrators 
calculate targets for each subgrantee. An IHE administrator in one state commented that he 
would like to see “not just what the formula is, but how the state got the numbers for each 
institution.” A local CTE director in another state expressed frustration about the lack clarity 
regarding the negotiation criteria used by state administrators: “It has not been made clear 
what [the criteria are for] them to accept the negotiating rate. If I came in and said, “Well, I 
want 72 percent as we get farther down the road,’ I’m unclear as to how they would make a 


determination of whether to accept it or not.” 


An IHE administrator in another state noted the disconnect between performance targets 
and local plans. “The rigor of reporting on the objectives of the local plan never matched the 
rigorous data for the performance measures,” he observed. “POS is a good example. You 
develop a local plan with a big emphasis on developing POS . .. but you’re not asked to re- 
port any hard data on that, like we do on these other measures. The plan often doesn’t deal 


with improving performance measures that we have to report.” 
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Use of Local Performance Data 
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Perkins administrators in most case study states reported using local performance target data 


to identify programs in need of improvement. In five states, administrators specifically men- 


tioned flagging LEAs that were below 90 percent of any performance target (as specified in 


Perkins IV). An administrator in one state described a color-coding system with green, yel- 


low, and red report flags in the state’s database to show “whether or not locals met their tar- 


gets, were within 90 percent of their targets, or were [within] less than 90 percent of their 


targets.” 


Several state administrators mentioned 
program improvement or corrective ac- 
tion plans that are required of local sub- 
grantees that fail to reach the 90 percent 
threshold for any indicator. For exam- 
ple, a state-level professional develop- 
ment director outlined a “risk-based 
monitoring” process in which LEAs 
failing to meet 90 percent of a negotiat- 
ed performance level are subject to 
closer monitoring and required to write 
a corrective action plan focusing on the 
core indicators on which they came up 
short. A secondary administrator in an- 
other state said that “a lot of our school 
districts” are putting Perkins in their 
broader school improvement plans, but 
she also noted that districts questioned 
the performance reports generated from 
the state’s database. She explained, 
“There are quite a few school districts 
that actually appealed their performance 
on 4S1 [student graduation rate] be- 


One State’s Accountability 
Hierarchy 


A state CTE administrator described the hierarchical scor- 
ing system used to identify programs in need of improve- 
ment: 


Any agency scoring at or above 90 percent, they'd be 
compliant. If an agency is below 90 percent in one [of its 
annual performance targets], it is classified [as] Needs 
Improvement. Agencies scoring below 90 percent in three 
or more of [their] approved annual performance targets or 
scoring below 60 percent of any single annual perfor- 
mance target are considered priority improvement agen- 
cies. Agencies scoring in the lowest percentage of overall 
performance, as determined by composite ranking of all 
performance level measures, will be considered monitored 
agencies and will be subject to monitoring visits. This year 
we're looking at the bottom 10 percent only because we’re 
limiting travel, but really we want to look at the bottom one- 
third. This is the first year we’ve used Perkins data as a 
determinant on who gets a [monitoring visit]. Priority im- 
provement districts are to complete a self-assessment tool 
that’s aligned with the 11 elements from our state plan and 
then, if they have found problematic areas, they can re- 
quest technical assistance from us. 


cause they didn’t think that the data were accurate and encompassed all of their CTE.” 


The state CTE director in another state reported that a majority of secondary Perkins sub- 


grantees will be cited this year and required to develop program improvement plans. He also 


indicated that performance data will become more useful for identifying programs needing 


improvement after the full complement of data on all indicators is collected and analyzed, 


something that “didn’t happen in the first year.” A local director in this state, however, cited 


some challenges associated with developing those plans: “This was actually the first time this 


year where the state gave us data and said, “These pieces of data are deficient. You need to 
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write a plan around making the data better.’ And then our turnaround is, ‘We had no control 
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over these data.” He explained that students take the assessment used for the Perkins aca- 
demic indicator in the fall of their junior year, which also is when students typically enter ar- 
ea CTE centers: “So they’re taking [the test] once .. . and they’re never tested that way again. 
... Not even the high school could tell you if those kids improved or not, and we’re being 


asked to write improvement plans around those data?” 


A postsecondary administrator in another state agreed that the 2009-10 school year will be 
“the definitive year” for local accountability reporting. She summarized what she learned in 
compiling the state’s Consolidated Annual Report and how she anticipated using the infor- 
mation for program improvement: “There were a couple of colleges that I identified that 
were having trouble kind of across the board, and I was looking for some patterns. They 
tended to be some small towns, rural, and so that’s something that I want to investigate fur- 
ther. Pll be able to say, ‘What’s going on that you’re having trouble with all of these indica- 


tors? You either missed them, or you almost missed them a couple of years.” 


Some state administrators cited the importance of working with local subgrantees to ensure a 
common understanding of the data. A state secondary administrator said, “We can’t do 
[program improvement] in isolation from the locals. So understanding the strategies they’re 
going to use and the recommendations that we might make, you pull those together, then 
that’s where we see the improvements that are going to take place.” A postsecondary admin- 
istrator in another state described how he plans to work with staff at individual IHEs to en- 
courage review and use of Perkins data. He noted that some IHEs will meet the negotiated 
target for the nontraditional indicators, but the disaggregated data will reveal problems in the 
performance of some subgroups. “We even make them count how many of these special 
population groups are below their district negotiated targets in their programs,” he ex- 
plained. “We could do it electronically, but we make them count just so we know they 
looked.” 


In several states, local practitioners reported limitations in using performance data received 
from state officials. One local CTE coordinator talked about how accountability data cannot 
help her local leadership team identify programs in need of improvement, because the data 
are not disaggregated by school or program. She said that disaggregated data “would be 
mote informative to me than ‘Here’s how all of your CTE kids are doing.’ ... We can brain- 
storm as a group, but having [data] disaggregated by CTE program would be much more 
valuable.” A high school principal in another state echoed her concerns: “The report doesn’t 
identify [the data] down to the school level or the program level,” he said. “It tells you the 
local target, percent increase, total increase required to meet the target, and that sort of thing 
... but it’s very difficult because it’s not specific to the program. ... You might have one 
program that has one completer, you might have another program that has 20, and no way is 


that broken down for you to make anything out of it.” 
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Local administrators offered varied opinions about the extent to which local performance 
targets actually led to improved performance. One local CTE director remarked, “I don’t 
think there’s been any big impact because we’re reviewing all students, whether they’re in a 
CTE program or not.... We are data-driven. We progress-monitor our kids to death proba- 
bly... but we are constantly looking at data and determining what’s our action plan and 
what are we going to do with those students.” A counterpart in a large district in another 
state suggested that the reporting standards “raise the bar” for CTE programs but indicated 
that local staff would be looking at the data anyway: 


I think the fields in career and tech education are only as strong as the 
weakest link. ... We were going to do it anyway, but I don’t want to trust 
that somebody else is going to do it and have it negatively impact the pro- 
grams. So, in that sense, I think anything the legislation does to .. . create 
strong standards for everyone is a good thing. It’s a headache, but it’s not a 
bad thing. 


State and local CTE administrators offered mixed views on whether negotiated local per- 
formance levels should be maintained in future reauthorizations of Perkins. One state-level 
administrator questioned the value of negotiating with individual LEAs and IHEs. “It’s very 
bureaucratic,” she observed. “The bottom line is that it takes a lot of staff time to do it. If 
the state is being held accountable for a target, ... I don’t know why we would differentiate 
the performance in our local levels for something that we expect everyone to be doing.” She 
also expressed concern that the negotiation process could delay funding awards in the future: 
“If youre going through this analysis process and they don’t want to accept your recom- 


mended adjustment level, we can be at this game for a long time. To what benefit?” 


A local CTE director in another state supported the concept of local performance targets 
but insisted that those targets should be appropriate for each state. “Hold us accountable for 
things that we have control over,” he said, noting that indicators appropriate in comprehen- 
sive high schools or four-year technical high schools are probably not appropriate for use in 
a shared-time, half-day model. “It makes you wonder, on a federal policy level, is it possible 
to design accountability that is not just one set of accountability requirements? Can the ac- 


countability system be adjustable based on the kind of model that’s offered in the state?” 


Sanctions for Inadequate Performance 
Perkins IV requires states and local subgrantees to develop and implement program im- 
provement plans if they fail to achieve at least 90 percent of the performance level for each 
core indicator. It also outlines a sequence of additional sanctions, including possible loss of 


Perkins fanding, for those that fail to improve performance. 
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Several state officials expressed concerns about the use of sanctions at the local level and 
questioned whether they would lead to improved local performance. One postsecondary 
state CTE administrator bluntly asserted that sanctions could lead to less oversight of local 
CTE programs: “When it was about program improvement and not sanctions, give me a cat- 
rot and I can lead them on. Give me sanctions, and they just don’t want to participate any- 
more, which means that I have less influence over what they’re doing in their programs 
because they don’t want my money.” A secondary colleague emphasized steps taken by state 
staff to limit the need for sanctions. “What we want to do is provide .. . the help and assis- 
tance you need so you’re not sanctioned,” he said. ““That’s why we set up this data monitor- 


ing process and are helping the locals clear up their data.” 


A Perkins administrator in another state also emphasized the importance of working with lo- 
cal subgrantees to improve performance. She reported that state administrators had men- 
tioned the possibility of sanctions but had yet to institute a formal sanctions process: “What 
we said is that, rather than taking money away, which doesn’t make sense if you’re struggling, 
is that we would have them focus their funds. ... They wouldn’t lose those Perkins funds, 
but they would be obligated to spend either all... or a portion... on improving the per- 
formance data that are falling behind. They’d lose that discretionary opportunity.” 


A secondary administrator in a third state acknowledged the potential power of sanctions as 
a program improvement tool but emphasized that local subgrantees are expected to shoulder 
significant responsibilities for Perkins accountability data. “In the grand scheme of things, 
[Perkins funding] is not very much money for a number of the districts,” he observed. “For 
them to go through the process of submitting data, following through with all of the other 
assurances and requirements associated with it, and then be flogged publicly and told you 
can’t use your money for this particular purpose without fixing something, it undermines the 
process.” A counterpart in another state used similar language: “We want to move down the 
road to more of a performance-based carrot approach, in that you’re rewarded for doing well 
as opposed to smacking people over the head or somehow publicly flogging people for not 


doing well.” 


Although most state administrators cited the 90 percent threshold as a trigger for an im- 
provement plan, several also described longer-range strategies based in part on the lag time 
in Perkins reporting. As one secondary administrator observed, “We’re looking at a multi- 
year process to identify whether or not someone really has a problem. That’s the real key— 
some sort of additional intervention, other than having the right program improvement 
[strategy], would be required to help if they’re seriously deficient.” Noting that state CTE 
team members believe the Department should review performance based on three years of 
data, she cautioned that they have not seen enough years of local Perkins IV data to identify 


real problems and underscored the possibility of considerable variability from year to year. 
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An administrator in another state (A) cited challenges associated with the timing of the man- 
datory program improvement plans, which are supposed to be developed and implemented 
the year after subgrantees fail to meet performance targets. “It doesn’t make sense to have 
subgtantees put together plans and have only six months to influence data that get reevaluat- 
ed the next year,” he explained. “So we have indicated that it’s a two-year plan and that they 


have to be working toward the targets in the first year.” 


A postsecondary administrator in a third state noted that local subgrantees would not be 
subject to directed funding requirements until they miss performance targets for three con- 
secutive years. She also indicated that state officials would require IHEs facing these sanc- 


tions to include budget lines for program improvement in their plans for the next year. 


Officials in two states reported that they had developed specific plans for financial sanctions, 
and one reported already using them. In one of these states, the CTE director said that sub- 
grantees failing to attain performance targets for two consecutive years “will no longer be in- 
cluded as CTE programs eligible for continued Perkins funding.” In the other state, a 
postsecondary administrator reported withholding funds for an institution that “did not 
complete the purpose of its grant .. . or spend all of its money.” She also emphasized that 
the nontraditional indicators are the only ones for which she anticipates more widespread 
problems with reaching targets, primarily because small populations increase the potential 


for percentages to be inaccurate and unreliable. 


Administrators in two states identified data quality issues as a potential barrier to the use of 
sanctions. “Occasionally ... we have a data reporting problem that has to get fixed that may 
have affected performance,” a secondary administrator acknowledged. In the other state, a 
postsecondary administrator reported a programming error in the state data system that led 
to the suspension of sanctions from last year: “Even though there were sanctions last year 
and people wrote to them and planned for them, there will be no second-year sanctions this 


year. We’re starting as if this was the first year again.” 


At the state level, CTE administrators reported relatively little progress on implementing re- 
ward structures for high-performing Perkins subgrantees; only one of the six states had a re- 
ward process. In this state, some postsecondary subgrantees were eligible to apply for an 
additional $150,000 in their competitive grants if the “program warrants it and is large 
enough in scope.” Administrators in at least two other states expressed interest in offering 
rewatds or moving to performance-based funding in the future. As one state CTE adminis- 
trator remarked, “Our ultimate goal is to reach a point where we can provide performance- 
based funding [for] school districts and/or community colleges.” In another state, a CTE 
administrator acknowledged, “We haven’t talked rewards; we don’t have the funds for re- 


wards.” 
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Local respondents generally were able to describe potential sanctions and the types of per- 
formance deficiencies that would trigger them, but some questioned the targets that could 
lead to sanctions. For example, one postsecondary administrator suggested that sometimes 
the performance goals “are just not realistic for certain populations or certain regions of the 
state, and yet we’re all beholden to those reporting efforts.” A local CTE director in another 
state indicated that he should have negotiated more appropriate performance targets: “I told 
them we wouldn’t be able to meet the goal. The next time around, I will not let them put us 
in that position, because it’s negotiated. ... We are a low, low-performing district. When you 


look at the students with whom we’re working, I knew we couldn’t meet that goal.” 


Local CTE administrators in some communities were concerned about the possibility of fi- 
nancial repercussions ftom their Perkins accountability data. One reported that “You get a 
sanction ... and a decrease in funding if you miss your targets for two years in a row.” A 
counterpart in another state indicated that Perkins funding was tied to improvement plans: 
“If we don’t write the plans, we won’t get next year’s Perkins money. So there is a money 


sanction tied to it.” 


Other local practitioners were less critical of the prospect of sanctions. “I don’t like the 
thought of potentially losing funding if you didn’t make one of your outcomes,” one IHE 
administrator acknowledged. “But .. . if you don’t meet one, you get time to submit an im- 
provement plan. ... It’s strict, but reasonable.” A colleague in the same institution was less 
clear about when sanctions might be applied: “In theory, I think we know that if we haven’t 
met the measures, then you’re supposed to develop an improvement plan. If your improve- 
ment plan doesn’t lead to improvement in the measures, then you can face a sanction. We 


didn’t meet our placement target last year. I’m waiting to see if there are sanctions to that.” 


A local secondary CTE director in that state was more philosophical about the possibility of 
sanctions. “I have heard of sanctions,” he said. “But if we lose $150,000, we will continue to 
do good things.” His counterpart in a larger district reported that the state CTE director did 
not anticipate local sanctions in the foreseeable future: “My impression is that there’s more 


pressure on the state with the feds than there is with us with the state in terms of sanctions.” 


One local administrator actually ascribed benefits to the local performance targets and pos- 
sible sanctions, suggesting that they would apply only to suberantees that “underperform 
consistently.” Perkins “is a program that makes us feel good,” she explained. “When you 
don’t have to make any excuses about why you’re doing so poorly because you met the crite- 


tia, you feel pretty good.” 
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Implementing Perkins !V Programs of Study Provisions 


Perkins IV required LEAs and IHEs to offer “career and technical education programs of 
study” (POS). This section describes the status of POS implementation in case study states 
and communities. Particular emphasis is placed on the processes used to develop, approve, 


and implement POS and the challenges encountered in these efforts. 


Major Funding Provisions in Perkins IV 


Perkins IV introduced POS to increase coordination within the CTE system, strengthen inte- 
gration of academic and technical instruction, and enhance connections between secondary 
and postsecondary education. As defined in Sec. 122(c)(1)(A) of Perkins IV, POS 


e incorporate secondary education and postsecondary education elements; 


e include coherent and rigorous content aligned with challenging academic stand- 
ards and relevant career and technical content in a coordinated, nonduplicative 
progression of courses that align secondary education with postsecondary edu- 


cation to adequately prepare students to succeed in postsecondary education; 


e may include the opportunity for secondary education students to participate in 
dual or concurrent enrollment programs or other ways to acquite postsecondary 


education credits; and 


e lead to an industry-recognized credential or certificate at the postsecondary level 


or an associate’s or baccalaureate degree. 


Perkins IV requires states to address how they would develop and implement POS as part of 
their Perkins state plan submitted to the Department for approval. Local subgrantees are re- 
quired under Perkins IV to offer one or more POS. In early 2010, the Department released 
its Programs of Study (POS) Design Framework (Framework), which consists of 10 components 
that expand upon and clarify the four minimum components of POS identified in Perkins 
IV. 


APPENDIX C. CROSS CASE REPORT 


10. 


POS Design Framework 


Legislation and Polcies—Federal, state, and local legislation or administrative 


policies that promote POS development and implementation. 


Partnerships—Ongoing relationships among education, business, and other 
community stakeholders that are central to POS design, implementation, 


and maintenance. 


Professional Development—Sustained, intensive, and focused opportunities for 


administrators, teachers, and faculty involved in the design, implementation, 


and maintenance of POS. 


Accountability and Evaluation—Systems and strategies to gather quantitative 


and qualitative data on both POS components and student outcomes to aid 


ongoing efforts to develop and implement POS. 


College and Career Readiness Standards—Content standards that define what 
students ate expected to know and be able to do to enter and advance in 


college and/or careers. 


Course Sequences—Nonduplicative sequences of secondary and postsecond- 
ary courses within POS that ensure that students can make a transition to 
postsecondary education without duplicating classes or requiring remedial 


coursework. 


Credit Transfer Agreements—Credit transfer agreements that provide opportu- 


nities for secondary students to gain postsecondary credits, supported by 


formal agreements between secondary and postsecondary partners. 


Guidance Counseling and Advisement—Guidance counseling and advisement 


that help students make informed decisions about which POS to pursue. 


Teaching and Learning Strategiee—Innovative and creative instructional ap- 
proaches that enable teachers to integrate academic and technical instruc- 
tion and students to apply academic and technical learning in their POS 


coursework. 


Technical Skill Assessments—National, state, and/or local assessments that 


provide ongoing information on student attainment of the necessary 


knowledge and skills for entry and advancement in postsecondary education 


and careers in their chosen POS. 
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While POS is new under Perkins IV, the idea draws on previous CTE reform initiatives such 
as Tech Prep and curriculum integration. Tech Prep programs under Perkins III were ex- 
pected to combine at least two years of secondary and two years of postsecondary education 
in a sequential, nonduplicative course of study. This “2+2” design, founded on articulation 
agreements between LEAs and IHEs, was intended to help students acquire academic 
knowledge and technical skills by integrating academic and technical instruction and to lead 
them to an associate or baccalaureate degree or a postsecondary certificate in a specific ca- 


reet field. 


Key Findings 


State CTE administrators in most states indicated that they incorporated the four statutory 
POS components into technical assistance efforts and approval criteria used for local POS. 
Although most state administrators expressed opposition to regulatory language defining 
POS, a number said that the Department’s Framework would have been helpful earlier in 

the process of POS development and implementation. At both the secondary and postsec- 
ondary levels, perceptions of POS varied considerably: some CTE officials identified the 
POS concept as the most important change in Perkins IV, while others viewed it as simply a 
new name for an existing approach. Key findings related to the implementation of Perkins IV 


funding provisions are summarized below: 


e Processes and policies for POS development, approval, and implementation 
vatied across and within states—and most CTE administrators advocated for 


continued flexibility in these areas. 


e Development of effective partnerships between the secondary and postsecond- 
ary systems was a major hurdle for many states, LEAs, and IHEs, even though 
there was widespread understanding that coordination between the two systems 


was a major goal of Perkins IV. 


e Relationships and processes established under Tech Prep played a major role in 
POS development and implementation in many locations, regardless of whether 


the state had merged Tech Prep and basic grant funds. 


e State and local CTE administrators were rarely able to provide data on POS en- 
rollment or outcomes, and none reported systemic capacity to track students 


from secondary to postsecondary education. 


e Most barriers to POS development and implementation were embedded in the 
institutional structures and organizational cultures within which people were 
developing POS, reflecting the fact that POS, by definition, attempt to bridge 


divides such as the traditional separation of CTE and academic studies. 
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State Support for Federal Guidance on POS Components 


Most state CTE staff interviewed said that federal guidance would have been helpful earlier 
in the process of their development and implementation of POS. Several specifically praised 


the quality of the Department’s Framework, however. As one state CTE director comment- 
ed: 


The work that was recently done out of [the Department] around programs 
of study, I wish we would’ve had that three years ago. That should’ve come 
out immediately. We wouldn’t have the mess we’re in right now [with the] 
diversity of [approaches in different] states. ... So it’s going to be more dif- 
ficult now, because states have already gone down certain roads. We’ve got 
to figure out how to create some national structures that make sense. I’m 
not a big proponent of the federalization of education. However, there are 
components that just make sense [so] that we can talk the same language 


and look the same in some ways. 


Another state director asked for further guidance about “exactly how we ate to use that 
framework. Are we going to be accountable for that, or is that just simply a tool that we use? 


I would say some additional guidance would be needed.” 


A few state officials said they preferred no federal guidance under any circumstances, while 
staff from four states expressed mistrust of regulatory guidance or statutory language that 
would be used to define POS. As one senior state administrator noted, “TI think if federal 
regulations were going to happen for POS, it needed to happen at the time it was rolled out. 
I think that the cow is out of that barn. To call it back now and say, ‘Oh, no, you’re doing it 
all wrong.’ I don’t think you can go back and do that and have it an acceptable way of doing 
business.”’ Another observed, “Having structure and a framework isn’t bad, [but] requiring 


us to do certain strategies strips the innovation out of it.” 


Several CTE administrators expressed a philosophical objection to guidance from the De- 
partment, opposition that was based on their perspectives for the appropriate roles for state, 


local, and federal governments in CTE. As one state administrator observed: 


The problem with regulatory [guidance], in my opinion, is who provides in- 
put to make decisions? . . . If they do something on the national level to get 
the practitioners involved in deciding what the regulatory guidance is, I 
don’t have a problem with it. But if it’s just somebody up there in the Belt- 
way, who may or may not have been in a CTE program or hasn’t been in 
one in a long time or has their own vision of what it is, then I think you 


have a hard time trusting regulatory guidance without appropriate input. 
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Support for Federal Guidance on POS 


State officials in several states described the types of specific 
federal guidance they would support: 


[Some states seem to think POS is just about] organizing CTE 
curriculum . . . that’s why we needed that guidance, needed 
the push to say: . . . "This is about the whole alignment of sec- 
ondary and all these components.” (State CTE director) 


What are the outcomes you want from a program of study? .. . 
Congress could save time, money, and paper by looking at 
what they really want from the postsecondary, focus the legis- 
lation on those few items [or] outcomes, and then allow states 
the flexibility to mount possible solutions to attain those out- 
comes. (State CTE administrator) 


POS [should be tied to accountability measures.] . . . having 
some clear guidance from [the Department] in terms of the 
measurement approach that we use for [POS] would also be 
very helpful. (State CTE director) 


Maybe the new legislation will better refocus the system, be- 
cause | think you've added and added, . . . but the money has 
stayed pretty much the same. So you’re watering down the 
soup, and at some point a refocusing of the program is proba- 
bly going to be necessary. (State CTE administrator) 


A representative from another state 
observed: “I don’t like rules coming 
down from the top. . . . Small states 
want regulation. They think that they 
can influence their legislature. Our 
legislature could care less what Perkins 
requires. Just because [the Depart- 
ment] requires something in regula- 
tion doesn’t mean that they are going 


to follow that path.” 


Administrators in several states noted 
that one positive result of less strin- 
gent direction from the federal level is 
that it encourages buy-in and collabo- 
ration. As one said, “We have close 
partnerships with our other agencies, 
and we work together to figure it out. 
... If it had been mandated and there 
had been regulations governing it, 
what we have may not be as effective 
as it is, because there wouldn’t be the 


ownership that we have now.” 


A few state administrators said they did not favor regulatory guidance on POS because regu- 


lations from the Department on other statutes had not taken into account differences across 


the states. As one explained, “If they could take into account rulemaking that would treat 


small states and large states, rural schools and urban schools, shared time regional systems 


and four-year comprehensive technical high schools . . . with equity somehow. I’ve been do- 


ing this 30 years or something and so far I haven’t seen intelligent rulemaking come along 


too often.” 


Developing and Implementing POS: State and Local Progress 


An important goal of Perkins IV is “increasing State and local flexibility in providing services 


and activities designed to develop, implement, and improve career and technical education, 


including tech prep education” [Sec. 2(3)]. This flexibility was evident in the processes for 


development and implementation of POS, which varied widely across and even within states. 


All state administrators reported that every local subgtantee offered one or more albeit ru- 


dimentary) POS, because this was a condition for receiving grant funds and a part of the lo- 


cal plan approval process. In contrast, administrators in at least one LEA acknowledged 
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they were not yet offering POS, and other local practitioners described offerings that fell 


considerably short of the required components. 


State and Local Definitions of POS 
Guided by the four components of POS outlined in Perkins IV, most state CTE officials in- 
dicated that they defined POS precisely as the term is defined in the law. Several states in- 
cluded a minimum number of credits or hours of secondary coursework to be considered a 
“sequence” of courses in their definition. Although there seemed to be an understanding in 
all states that POS are meant to incorporate elements of secondary and postsecondary edu- 
cation, there was a broad tendency to focus on the secondary side of POS. This focus oc- 
curred despite states’ adoption of POS definitions that emphasized links to postsecondary 
education. A state administrator in one state, for instance, defined POS as “a sequence of 


courses leading to postsecondary education.” 


In four of the six states, POS were seen as leading to—trather than inclusive of— 
postsecondary education. A staff member at one postsecondary state agency explained, “We 
define secondary POS and postsecondary POS differently.” State administrators in only one 
state emphasized the coordinated, nonduplicative secondary-postsecondary aspects of POS 
when asked to define the term. The disconnect between the secondary and postsecondary 


sectors in how they defined POS was also readily apparent in other states. 


States communicated their definitions of 


Local POS Definitions 


eral ways. These included the state Per- Local secondary practitioners explained their 
kins plan, statewide and local confer- understanding of POS: 


POS to local subgrant recipients in sev- 


erees ane meet hes isis by. state tech Working with the local postsecondary insti- 


nical assistance providers to schools and tutions, so that graduates from your pro- 
districts, and print and web materials. It gram have either completed some college- 
level work before they graduate and/or go 
on to a related course of study at the post- 
secondary institution. 


was evident, however, that state defini- 
tions of POS were not always well un- 
derstood at the local level. Local defini- 
e A [program of study] is a student roadmap, 
but it also helps the teacher . . . to keep that 


vision that what they’re doing can extend to 
tions, but with significant variation, and, the bigger picture. 


tions of POS generally reflected at least 


some part of the federal and state defini- 


in some cases, confusion and ambieuity. 
guity e Our [POS] here [are] is the course listings 


and descriptions provided for students and 
ferred to secondary-postsecondary con- parents to look at as they choose classes 


nections in their POS definitions, with and programs [in which] to enroll. 


an equal number noting a rigorous se- 


More than half of local subgrantees re- 


quence of courses. 


APPENDIX C. CROSS CASE REPORT C-99 


Less than one-fourth of local subgrantees explicitly mentioned dual or concurrent enroll- 
ment in their POS definition. A postsecondary administrator in one state said, “I’m not sure 
if dual enrollment is actually part of the POS. It’s very confusing what is and is not part of 
the POS.” Nearly a third defined POS as leading to an industry certification or degree. One 
local CTE director could not recall receiving a POS definition from the state but appreciated 
“the academic and professional freedom to grow and make mistakes.” A postsecondary ad- 
ministrator in the same state said, “I don’t think there’s a written-down statewide definition 
of POS in place.” 


Some confusion or ambiguity in defining POS at the local secondary level may be related to 
use of the “programs of study” terminology, which in some districts was already in use be- 
fore Perkins IV to refer to something entirely different, such as a course catalog. In many 
cases, local administrators defined POS in relation to their understanding of “pathways,” 
“career pathways,” or “career clusters.” Some of these definitions were based on the 16 ca- 
reer clusters of the States’ Career Clusters Initiative,> while the general understanding of 
POS in some communities appeared to be based on a mote localized or informal conception 
of an industry- or career-themed sequence of courses. Administrators and faculty in many 
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LEAs used the terms “pathways,” “career academies,” and “POS” with little distinction. A 
staff member in one district described POS as “a tool that we use as a career pathways pro- 
gram to make certain that each of us in the process is doing what we’re supposed to be do- 
ing.” Many local officials seemed to view POS as the paper manifestation, in the form of a 


completed template, of a career-themed sequence of courses. 


Some local secondary respondents expressed the view that POS were “nothing new” but ei- 
ther an expansion of an existing concept or merely “new language” applied to an old idea. In 
four states, this view was espoused by one or more respondents in at least one local com- 
munity; in another state, this view seemed to prevail in all local districts. According to an 
administrator in a small district in this state, POS were “just sticking another title to [the 


term] and seeing how it fits into some kind of jigsaw puzzle that somebody came up with.” 


State Support for POS Development and Implementation 
Despite the relatively small amount of Perkins funds in the context of state and local educa- 
tion budgets, some state CTE directors said they attempted to use Perkins strategically as a 


lever for change in CTE. According to the CTE director in one state, “It’s a generational 


3 A career cluster is a grouping of occupations and broad industries based on commonalities. The 16 ca- 
reer clusters provide an organizing tool for schools, small learning communities, academies, and mag- 
net schools. The 16 career clusters are Agriculture, Food, & Natural Resources; Architecture & 
Construction; Arts, A/V Technology & Communications; Business Management & Administration; 
Education & Training; Finance; Government & Public Administration; Health Science; Hospitality & 
Tourism; Human Services; Information Technology; Law, Public Safety, Corrections & Security; 
Manufacturing; Marketing; Science, Technology, Engineering & Mathematics; and Transportation, 
Distribution, & Logistics (http://www.careerclusters.org/list1 6clusters.php). 
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change here, and we’re hanging program|s]| of study out there as the banner that we’re going 
to change things under, but we’re getting it from both sides.” Similarly, the state director in 
another state described POS as “the most important thing in the Perkins legislation” and part 
of a larger vision for high school redesign around pathways. The use of Perkins as a catalyst 
for change was noted in a third state, whose state director said, “When we talk about school 
reform, actually Perkins over the last few years has provided us with some resources to be- 


come a driver [of change].” 


State staff have provided many forms of leadership, support, and technical assistance to local 
suberantees as they developed and implemented POS. While there were some common cat- 
egories of support, the intensity of support varied somewhat according to state agencies’ ca- 
pacities and readiness to support POS and state-level legislation and policies (or lack thereof) 
related to CTE. 


State Capacity and Readiness 

States exhibited varied capacity to support POS development and implementation, and staff 
across the states expressed diverse views on whether the Department had provided the in- 
formation and technical assistance they needed to support POS efforts in LEAs and IHEs. 
State agency staff in two states indicated that they had anticipated the requirements of Per- 
kins IV and laid some preparatory groundwork for implementing POS. An agency official in 
one of those states said, “We saw it coming. We were out there doing workshops and telling 
districts way before Perkins IV was even authorized that this is coming down the chute, and 
this is the way it looks like it’s going to happen.” By contrast, a staff member in another state 


described staff at the Department as being “very guarded about what they tell us.” 


State administrators in several states explained that previous work on Tech Prep under Per- 
Rins II] and an understanding of related ideas like the career clusters provided advantages in 
developing POS. In some states, however, the short turnaround time required for POS im- 


plementation after passage of Perkins IV was described as a source of frustration. 


CTE directors in several states commented on the Department’s release of the Framework 
in early 2010. Arriving three years after the passage of Perkins IV, the Framework was viewed 
by most state CTE directors as useful but tardy information. As mentioned previously, many 
state administrators stated that earlier guidance would have been helpful. As a state agency 
technical assistance provider explained, “It seemed like there wasn’t a complete vision at the 
federal level, as far as what they really expected and wanted. Yet they were expecting us to 


implement something that they didn’t really know what it was.” 


States’ capacity to support POS development and implementation has also been undermined 


by budget constraints and staff reductions. In one state, budget difficulties had reduced staff 
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at the state department of education to the extent that only one staff member remained to 


provide services to local subgrantees. 


State Leadership and Policy Guidance 
State-level legislation and policies related to POS specifically and CTE in general varied 
greatly and affected administrators’ abilities to influence POS development and implementa- 
tion. In one small state with no state CTE rule or statute, state agency staff relied heavily on 
Perkins to promote new CTE initiatives such as POS. In contrast, an official in a state where 
the eligible agency played a central role in POS development statewide reported that CTE is 
“embedded in... our laws [and] in our department policies.” In a third state, the legislature 
had passed CTE legislation and the current gubernatorial administration was seen as sup- 
portive of CTE, but there had been no effort to align this state legislation with Perkins or en- 


act state law on POS. 


While many state administrators relied on Perkins funds to promote change, they met re- 
sistance from some local subgrantees when attempting to implement policies not explicitly 
required by Perkins IV. One state director said, “These people [local subgrantees] have no 
problem picking up the phone and calling the Department and saying, “This is another un- 
funded mandate. I don’t have to do this.”’ Local resistance, in most cases, seemed to be 
based not in recalcitrance, but in the immense challenges of implementing a large-scale initia- 
tive like POS with some measure of replicability in a “local control” state. According to a 
veteran staff member in one state, “What we haven’t managed to really do is sell this idea 
that POS is the way to do it... to deliver CTE.” 


State agency representatives in five of the six states characterized their states as being guided 
by “local control” or “local determination” policies, whereas officials in the sixth state re- 
garded policy as more centralized. The implication was that state agency staff in local control 
states cannot mandate curriculum or the specific format of POS. As staff in one state ex- 
plained, “We would like to hope there was a typical process for POS development, but we 
are a state that is concerned about local determination and .. . there are some implications in 
a lot of divergent approaches to developing POS.” A lack of cooperation between secondary 
and postsecondary agencies at the state level also could complicate states’ efforts to deliver a 
consistent message on POS, as was the case in this state. By contrast, three state-level agen- 
cies in another state had cooperated on a statewide POS template and the delivery of tech- 


nical assistance on POS. 


Despite some resistance at the local level, state agencies in local control states were attempt- 
ing to exert some influence over POS development and implementation. As one state CTE 
director remarked, local control “doesn’t excuse an absence of state leadership.” The most 
common method for imposing some measure of consistency in POS development was the 


creation and dissemination of POS templates (either a single template or several templates 
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customized according to career cluster areas). State administrators noted some degree of col- 
laboration between secondary and postsecondary stakeholders as well as local input into the 
creation of these templates, but local perceptions of state responsiveness to local needs var- 
ied considerably. State agencies required local subgrantees to submit completed POS tem- 
plates with local plans, and, in most cases, the template could be modified to account for 
local needs. In the local control states, the templates were viewed as the clearest example of 


state involvement in POS development. 


Another approach commonly used by states in developing POS was the creation of “mod- 
el” POS, which LEAs and IHEs were encouraged to adopt. These model POS were devel- 
oped in different ways. In one state, for example, local consortia involving secondary and 
postsecondary education and industry competed for $100,000 grants to develop model POS. 
Another state had planned but not yet implemented an RFP process to award grants to area 
CTE centers statewide to develop model POS. To date, the state agency’s main role in this 
state had been convening statewide teams of CTE instructors and industry representatives to 
develop common capstone assessments for POS. An official in a third local control state 
said that the state is leveraging Perkins and POS to send a message to local districts and col- 
leges: “You will align your curriculum, and you will create the sequence of courses. And we 
don’t want students shot-gunning all over the place and then showing up at the postsecond- 
ary doorstep taking a bunch of remedial courses.” In all of these states, whether they relied 
on model POS or common capstone assessments, “what you see a lot of is encouragement 


to go in the right direction,” said one state official. 


In another state, administrators described a state-level partnership composed of secondary 
and postsecondary education and business representatives that was using the 16 career clus- 
ters to develop POS through a multi-year process of standards revision, curriculum frame- 
work development, and professional development. Local districts were encouraged to “plug 
in” their courses to the model POS. LEAs in this state had the option of developing their 
own POS, but those who did were unlikely to be approved. According to an administrator at 
the state eligible agency, “[Local districts] can create their own sequence of courses with their 
justification with everything that they put in, and we were pretty rigid on that. We held them 
to a high standard. Some were approved, [but] most were not, because they did not have the 


justification that made sense.” 
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Local Reactions to State-Level POS Development 


In one state where the state agency was leading POS development, local secondary and post- 
secondary officials expressed some dissatisfaction with the process: 


“Our frustration at the postsecondary level is [that the state education department] is setting this 
up. [The department] didn’t even talk to the advisory groups at the community colleges... .”A 
frustrated secondary administrator said, “What we thought we could get for our students and 
what they needed is different from what [the state agency is] telling us . . . there [are] some 
things that just didn’t meet up the way they should have.” 


Although leaders in another state also cited a local control system, the eligible state agency 
played a much larger role in developing statewide POS strategically in the career clusters that 
state staff considered most important to the state economy. This state released POS-specific 
templates for these strategic clusters that could be modified to fit local needs; local officials 
responded by looking to the state for guidance. Several dozen statewide articulation agree- 
ments were in place in this state, and there were plans to create more. Meanwhile, the state 
encouraged local subgrantees to go beyond the minimum of one or more POS and develop 


multiple POS suited to the local economy. 


The state role in POS development and implementation was markedly different and more 
centralized in another state. POS were developed by a statewide committee, divided into 
seven program areas and composed of local CTE directors, secondary and postsecondary 
faculty, state department of labor staff, business partners, teacher educators, national experts, 
CTE and academic consultants, and staff from the state eligible agency, and local districts re- 
lied on the eligible agency for guidance. There were nearly six dozen state-approved POS 
within the 16 career clusters and almost 50 statewide articulation agreements. Local percep- 
tions of this POS “menu,” likened by one high school counselor to a college catalog, were 
mixed, with different administrators in the same urban district describing it as “tremendously 
fragmented” and as “great.” Staff in a small district in this state described the difficulty of 
applying the state-designed framework locally because of limited course offerings and in- 


structional capacity. 


State administrators in four of the six states described how they were using Perkins funds to 
serve a third population, adult learners in noncredit CTE programs.‘ State officials in at least 
one of these states said that serving adults may present challenges for the POS concept, 
which they regarded as having been designed to work most smoothly within a model of four 


years in secondary education with a direct path into postsecondary education. 


4 Perkins allows state grantees to use funds “to provide career and technical education programs for 
adults and school dropouts to complete the secondary school education, or upgrade the technical 
skills, of the adults and school dropouts” [Sec. 135(c)(15)]. 
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State-Provided Technical Assistance 
State staff in all six states provided technical assistance to local subgrantees in a variety of 
ways. The most common form of technical assistance was conferences, held both statewide 
and regionally. At these conferences, state staff might walk local practitioners through the 
Perkins application process, explain the legislation, share best practices, or respond to specific 
questions from local subgrantees. Staff in one state indicated that they had provided more 


technical assistance under Perkins IV than under Perkins III. 


In at least half of the states, much of the technical assistance was directed toward helping lo- 
cal subgrantees complete and update their local Perkins plans. In several cases, staff helped 
local practitioners revise POS to meet state standards for approval. Administrators in two 
states specifically reported providing technical assistance on POS to guidance counselors. In 
one state, some local workshops focused on sequencing coursework in a specific career clus- 


ter. 


Depending on the size of the state, state staff might meet with local representatives as often 
as monthly or as rarely as bi-annually or annually. Administrators in one state agency held 
monthly conference calls with local subgrantees, alternating the calls between secondary and 
postsecondary CTE directors. Agendas for these monthly calls were based on questions 
submitted by staff in LEAs and IHEs; one of the most popular topics identified by local 


subgrantees was how to convert a local CTE program into a POS. 


Local respondents in all states indicated that state staff members were generally available to 
respond to specific queries on POS specifically or Perkins IV in general via phone or e-mail 
on a regular basis. Some larger states had set up a regional structure in which designated staff 
members were responsible for providing technical assistance and responding to inquiries 
from specific districts on POS and other Perkins-related topics. In many states, the eligible 
agency’s POS template was viewed by local subgrantees as a form of technical assistance 
from the state. Staff in one state shared completed POS templates with local subgrantees 


when high-quality products were submitted. 


In many states, travel budget cuts at the state level had curtailed state staffers’ ability to visit 
local subgrantees and provide on-site assistance. Commenting on the state’s education agen- 
cy, one local CTE director said, “They’re not a big department . . . and there’s only so much 
you can expect out of those few people.” Local recognition of the effects of the economy on 
state departments was widespread, but met with varying degrees of acceptance. In one state, 
a postsecondary administrator summed up his view by saying, “I’m sure they’re underfunded 
and have a lot of responsibilities, but we’re pretty much on our own.” According to a local 
secondary administrator in another state, “There was pressure to implement one [POS] but, 


besides that, little guidance.” 
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Local perceptions of the amount and quality of technical assistance provided by the state 
varied among local communities in the same state. While some local representatives were 
generally satisfied with the technical assistance provided by the state, others said it was insuf- 
ficient or not helpful to all audiences. Several factors might have contributed to this inequity, 
local officials suggested. In one state, a postsecondary administrator asserted that secondary 
officials had greater access to technical assistance than postsecondary officials. Assessing the 
quality of technical assistance, a secondary administrator in another state said, “To be brutal- 
ly honest with you, it all depends on the consultant [from the state agency]” and noted that 
technical assistance was sometimes undermined by personality conflicts. In some communi- 


ties, technical assistance from the state was seen as inadequate overall. 


State Processes for POS Approval 
All six states had adapted their application and review process for local Perkins plans to in- 
corporate POS submission and approval. At least three states had adopted the “language of 
the law” as criteria for approving POS. All or most states appeared to allow applicants to re- 
vise and resubmit their plans if they did not pass the first review. Beyond these commonali- 
ties, there was much variety in the approval processes, reflecting the diversity of CTE 


structures actoss the states (Exhibit C.17). 
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State 


Exhibit C.17. 
State processes for POS approval 


Process for approval 


Eligible local applicants submit POS for approval through an electronic portal along with the local 
Perkins application. Plans are reviewed by state CTE staff for compliance with the intent of the law, 
correct data, and alignment with assessment rubrics. 


Three collaborating agencies created a “program of study assurance,” a checklist for the minimum 
POS requirements in Perkins IV, that appropriate parties must complete before the POS is approved 
and Perkins funds are distributed. The statewide POS template must be attached to the curriculum 
and also submitted for approval. If appropriate paperwork is not submitted and there is no evidence 
of POS, applicants risk the loss of funding. 


Regional consultants at the eligible agency review local secondary Perkins plans to determine that the 
four required POS components are in place. The state approves the plan overall, rather than the 
POS separately, and the review process does not assess the quality of the POS. If applicants do not 
propose a viable POS, the state either provides technical assistance or declines to fund the plan. 


At the postsecondary level, the community college system approves programs, and the state postsec- 
ondary agency reviews the rigor of the courses offered. The system does not examine secondary 
courses. Before a course or ptogram reaches the state system office, it must be endorsed by the local 
curriculum committee and then one of seven regional CTE consortia. The system uses an online 
application requiring applicants to address POS requirements, including linkages between secondary 
and postsecondary levels, and explain how they target funds based on areas of need and expected 
outcomes. 


Local secondary and postsecondary subgrantees must state in their local Perkins plan how many POS 
they have developed and how many they will convert to the state-developed, cluster-specific POS 
templates. For approval, POS must be in high-skill, high-wage areas and based on local needs. The 
eligible agency reviews and signs off on each local application. If the applications are incomplete, 
there are discussions between the eligible agency and local subgrantees about revisions. 


The eligible agency has adapted the state program approval process into a POS approval process. 
Local applicants must submit a program design document identifying a cluster and pathway and 
demonstrating that the pathway is high-skill, and either high-wage or high-demand or both, with 
supporting labor market data. The document must also include an assessment plan, applicable indus- 
try certifications, a sequence of courses from grade 9 forward, and community partners, including 
postsecondary institutions, to be involved in POS development. 


LEAs ate required to identify a minimum of one or more POS linked to postsecondary education to 
receive Perkins funds. If state-developed POS are already in place, they are automatically approved by 
the state. If a LEA proposes different POS, it must follow standard criteria and obtain state approv- 

al. 


At the postsecondary level, each applicant must demonstrate at least one link with a secondary agen- 
cy as prescribed by Perkins IV, provide labor market data to support the demand, and include plans 
for sustaining the program. Applications are sent first to the state board of regents and then to the 
state higher education commission for acknowledgement and approval. 


Most state administrators said they allowed local subgtantees to revise and resubmit POS 


plans and were willing to provide technical assistance to help subgrantees get to that point. 


In one state, for example, administrators received 65 applications for Perkins IV Title I fund- 


ing in 2008-09; of these, only 25 were approved as submitted, but 40 were approved after 


revision and resubmission. According to state agency staff in this state, “It’s my responsibil- 


ity with the consortiums or schools that ’m assigned to, to get that [Perkzns] plan [including 
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the POS] approved, and if it’s wrong, then I work with them to make sure it is approved.” In 
the one state that had rejected several POS, submission of an inadequate POS plan could re- 
sult in rejection of the entire local Perkins plan. The state CTE director explained how some 


local districts had lost funding: 


A few districts lost funding because of the inability to get coherent POS in 
place; [they were] mostly rural, isolated districts. Local plans are reviewed by 
regional consultants at [the state agency] to determine that they have the 
various required components in place. If they don’t have [them], then [the 
state agency] doesn’t approve the application. And we get some really shaky 
ones in. Good example: people trying to use a CTE course sequence of 
marching band, jazz band, and choir. Well, that’s not a serious CTE course 


sequence. 


Local POS Development and Implementation 
Beyond the requirement that each local subgrantee offer one or more POS, Perkins IV does 
not prescribe a process for POS development and implementation. Consequently, consider- 
able variation was evident in how POS were developed at the local level. This section de- 
scribes local processes for POS development and implementation in accordance with the 


four POS components outlined in Perkins IV. 


Incorporating Secondary and Postsecondary Education Components 

While at least half of local respondents said that they understood that Perkins IV requires 
both secondary and postsecondary education components in POS, they also reported that 
promoting coordination between two very different institutions, namely LEAs and IHEs, 
was difficult. Local CTE officials were divided on whether leadership for POS efforts came 
from the postsecondary or secondary side, with slightly more respondents indicating that 
leadership had come primarily from IHEs. In one state, postsecondary leadership on POS 
resulted from a state directive, but in other states the process was guided by local decision- 


making. 


A lack of coordination between secondary and postsecondary agencies at the state level in 
one state was reflected in local programs. In the three districts studied in this state, there had 
been little attempt to create formal links between secondary and postsecondary institutions. 
State agency staff suspected that some secondary subgrantees had completed the postsec- 
ondary portion of the statewide POS template simply by looking at the local community col- 
lege’s course catalog and filling in plausible options for connecting coursework. In one rural 
district in this state, this indeed proved to be the case. In another community, the disconnect 
was so profound that the secondary Perkins IV coordinator made a call while researchers 
were present to ask someone to identify the Perkins IV coordinator at the local community 


college. 
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Local secondary administrators identified several barriers to effective secondary and post- 
secondary partnerships, including college officials’ unwillingness to work with secondary of- 
ficials, their lack of understanding of secondary programs, and geographic distance. At one 
site, the local CTE director reported that the reluctance of postsecondary faculty members 
to cooperate with secondary teachers posed a barrier to formalizing connections with K-12 


programs. 


Two postsecondary partners in another state had very different attitudes toward working 
with secondary agencies. One had established a staff position dedicated to working with sec- 
ondary partners, while the unionized faculty at the second institution balked at aligning their 
coutses with secondary offerings, despite their supportive administrators. In isolated rural 
districts in five of the states, distances between schools and colleges and poor weather and 
road conditions made access to postsecondary programs difficult. Community colleges in 
these rural areas also were likely to be serving many far-flung districts, posing an additional 
challenge to forming effective partnerships. One rural site was using distance learning and 


online learning to help the local community college reach a larger secondary audience. 


Less frequently, challenges in establishing secondary-postsecondary relationships were re- 
ported as resulting from unwillingness on the secondary side to work with a college with lim- 
ited program offerings. In one local district, the CTE coordinator recounted how one of its 
programs had been rejected for articulation by the community college because “there was 
nothing with which to articulate it.” He noted that a community college coordinator had vis- 
ited the local high school and told him that that he “should be teaching these [courses] at the 
college” —to which he responded, “No, you should be teaching something better so I can 


articulate with you. The kids can do this.” 


In many other local districts, lack of progress in effectively connecting secondary and post- 
secondary systems resulted not from unwillingness, but from the inability to overcome insti- 
tutional barriers and logistical problems with scheduling and transportation. In many of the 
local communities, the strenuous effort required to forge these connections, and the lack of 
time and resources on both sides of the secondary-postsecondary bridge, were frequent re- 
frains. As one postsecondary administrator explained, “I’ve got a full-time job as a dean, and 


I took this on. It’s difficult to spend the amount of time it would take to really do it right.” 


Success in forming effective secondary-postsecondary partnerships depended largely on rela- 
tionships between representatives of the two types of institutions at the local level and the 

commitment of individual administrators and instructors. Variation was evident, even among 
schools in the same district and between comprehensive high schools and area CTE centers, 


in the number and effectiveness of secondary-postsecondary partnerships. 
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In one state, articulation agreements were formed between individual secondary and post- 
secondary teachers, meaning that a retirement or relocation could end the articulated pro- 
gram. Another state required IHEs to demonstrate links with one or more local school 
districts to receive Perkins funding. Even in that state, however, respondents at one local site 
said the strength of these relationships varied. Communication across different types of sec- 
ondary institutions also was problematic; in several districts, comprehensive high schools 
were not aware of POS offerings available in area CTE centers and vice versa. Nevertheless, 
there were a few examples of secondary and postsecondary educators were working closely 


together to forge connections to benefit students. 


In one state, cohesiveness at the state level was 


Effective Collaboration 
between Secondary and 
Postsecondary Partners: 


mirrored by collegial working relationships be- 
tween secondary and postsecondary officials in 


the three local communities visited. In two of 


An Examp le these districts, secondary and postsecondary 
A local secondary CTE coordinator explained staff had been working closely together since the 
the rationale for collaborating with postsecond- 1990s. A high school principal in a large district 


ANE TSI SE (es in this state explained the cooperation by saying, 


Early on, way early on, it was about [secondary “What would be the opposite, a course in isola- 
instructors and postsecondary faculty] getting tion? That doesn’t make any sense.” The local 

to know each other, getting to understand this 
[POS] process if we’re going to work together. 


And in some ways we’re competitors, but in 
more ways we really are a team to make good POS. This more systematic, multi-institution 


community college there maintained a single 


multi-institution articulation agreement for each 


things happen for students, and when we re- approach to articulation also was being used in 
member that it’s always about the student, 
there [are] plenty of students and . . . plenty of 
ways to help students go through different 
pathways. 


local communities in another state, where previ- 
ous attempts at articulation had been “haphaz- 


ard.” 


Interagency cooperation and coordination in another state were described as modeling co- 
operation for local subgrantees. When submitting POS for approval, local subgrantees in this 
state “need to have a signed agreement from both secondary and postsecondary, along with 
their program of study template, so that [POS students] know [which college] the program is 
leading to... and which courses they need to take at the college level.” 


Many districts were also reaching down into middle schools or the earlier high school grades 
as they considered the future of POS. In one state, area CTE centers were beginning to offer 
“pre-tech” courses that would prepare 9th- or 10th-graders for entry into POS in 11th grade, 
while one district in another state provided a 10-week career exploration class for all 8th- 


graders (without using Perkins funds). 
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Alignment with Standards in a Nonduplicative Progression of Courses 
Administrators across the states were using a variety of strategies to align POS with state or 
national standards, but these efforts were limited in most states and local communities. Offi- 
cials in two states reported developing standards that were specifically attuned to career 
skills, not just to academic skills. A state administrator in one state reported, “All CTE pro- 
grams have frameworks that build in employability and leadership skills and are built to skill 
standards.” A secondary CTE instructor in this state recounted participating in a statewide 
committee to align CTE programs with core standards. The CTE program standards devel- 
oped in these statewide committees were available on the state agency’s website. State staff 
were unsure, however, how widely these standards were being used at the local level, and in- 
deed, a secondary instructor in another district in this state reported a lack of consistency in 
standards. “We're all over the place,” he said. “And that to me is frustrating.” Another state 
was developing a set of career-ready standards that include, according to a state administra- 
tor, “socio-emotional kinds of things about critical thinking and team work and responsibil- 


ity. 


> 


State directors in two states reported using assessments to drive alignment with standards. 
One state director described how the state was using nationally available third-party assess- 
ments in these efforts. The CTE director in another state reported efforts to innovate stand- 
ards-aliened exams that would be used across the state. Statewide committees in this state 
were developing common capstone assessments in four POS areas, which the director de- 
scribed as a two-year process of “validating the competencies and standards [in all POS] and 
then developing the assessment.” According to this state director, developing these common 
assessments “is the only way we have of honing in on common state standards for program 


ateas and common outcomes.” 


In several states, the state-provided POS templates reflected an effort to encourage local 
practitioners to conceptualize and codify nonduplicative course sequences that bridge sec- 
ondary and postsecondary education. One state required local Perkins applications to include 
a standardized articulation form that had been developed by the Tech Prep directors; four of 
the other states also required local applicants to list POS course sequences that encompassed 


both secondary and postsecondary elements on POS templates. 


In some cases, these efforts had begun well before Perkins IV. Officials in one LEA reported 
that they had started looking at core content areas and developing POS for students a few 
years before Perkins IV’ was passed. An official in another state said that Perkins IV “made us 
streamline our courses to make sure that they fit the new formula. We only offer those that 
can make a cluster.” A similar notion was expressed in a small district in another state, where 
the CTE coordinator explained a shift that came with Perkins IV: “T said, okay, I don’t want 


to articulate [individual] courses anymore. We articulate programs.” 
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Meanwhile, officials from small rural districts were likely to say that the requirement to align 
secondary and postsecondary education in the POS definition was not conceived with rural 
needs in mind. Rural districts often lacked the staff to offer the appropriate sequential cours- 
es, and postsecondary partners were likely to be relatively inaccessible. A secondary CTE di- 
rector in one state described the challenges facing rural districts: “I meet the requirements 
for having a [program of study] , but it is aligned with a college 90 miles down the road in a 


different county.” 


In addition to specific efforts to align POS with standards in nonduplicative course sequenc- 
es, officials in many LEAs and IHEs expressed a commitment to the broader goal of in- 
creasing CTP’s rigor, in part by integrating academics into CTE course sequences. These 
integration efforts might focus on integrating academic and CTE content into the same 
course, as was the case in one state, or they might focus on including one or more academic 
courses in the POS. For example, academic classes, including AP economics and personal 
finance, were part of the POS in one LEA. Staff at area CTE centers in two states also re- 


ported that they had academic teachers on-site to integrate academic and CTE instruction. 


Though there was still some of the old “vocational stigma” attached to CTE in several 
communities, many local administrators indicated that these innovations, along with inten- 
sive marketing efforts, were changing perceptions. According to a district CTE director in 
one state, “No longer is CTE seen as a dumping ground, but more of a proving ground.” As 
part of their efforts to integrate academics and CTE in coordinated sequences, a few local 
schools or districts had explored making CTE a graduation requirement, even where it was 
not required by the state. One district administrator explained, “We're trying to do a better 
job of integrating CTE with overall instruction. As we increase the number of credits re- 
quired for graduation, we'd like to have at least one credit of CTE required. This is part of 


out effort to integrate CTE with academics.” 


Not all respondents were comfortable, however, with fusing CTE and academic instruction. 
In a small community in one state, for example, a community college administrator ob- 


served, “I don’t even think the academic teachers understand where they fit into [POS].” 


Opportunities for Dual or Concurrent Enrollment 
Secondary and postsecondary educators in many local communities used the terms “dual en- 
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rollment,” “articulation,” and “dual credit’ —or some combination thereof— 
interchangeably. In many cases, they were not sure how each of these options was defined in 
the state. Based on descriptions from practitioners in these districts, either articulated credit 


or dual enrollment arrangements—or both—appeared to be in place in many communities. 


Articulation agreements between secondary and postsecondary institutions or individual faculty 


members at the two levels generally specified the conditions under which high school stu- 


APPENDIX C. CROSS CASE REPORT C-112 


dents might be eligible for postsecondary credit for a course taken at the high school. In dua/ 
credit arrangements, secondary students typically received both high school and college cred- 
its for successful completion of a college-level course. For dual enrollment, stadents were en- 


rolled concurrently in both high school and college. 


Some LEA and IHE officials acknowledged the weaknesses of articulation agreements under 
the Tech Prep system, especially the complicated processes usually necessary for students to 
obtain postsecondary credit. Findings from the 2004 NAVE report, The Structure and Challeng- 
es of Vocational Education Funding and Accountability Systems revealed several barriers to students’ 
ability to claim articulated credit under Tech Prep (White et al. 2004, p. 146). These barriers 
included requirements that a student complete additional courses or attain a specific score on 
placement tests to have the credits appear on the postsecondary transcript; the length of 
time between high school completion and college enrollment; and policies requiring students 
to identify the college-level courses they completed in high school and make formal requests 


for articulated credits. 


Some LEAs and IHEs had not yet developed effective strategies for addressing such chal- 
lenges, and the “portability” of credits remained a major stumbling block for the majority of 
local subgrantees. One postsecondary administrator described the process at her IHE: “The 
old model was [that] students would complete a [college]-approved course at the [area CTE 
center], and as long as [they demonstrated] by the time they graduated that they met college- 
level placement scores, then they could apply for the credit and it was retroactive. They had a 


couple-year window of time... . It was confusing and not effective. It didn’t make sense.” 


Although some local practitioners had come up with solutions, others had not devised ways 

to address the problems. Even when these arrangements were working, some LEA staff said 
they had a hard time convincing students and parents. According to one district administra- 

tor, “The biggest problem that we’ve had with dual enrollment is making folks really believe 

they’re getting the credit.” 


Some LEAs and IHEs had maintained articulation agreements established under Tech Prep 
and, in some cases, expanded or enhanced them or modified them into dual credit or dual 
enrollment arrangements. Most secondary-postsecondary partnerships revisited articulation, 
dual credit, or dual enrollment agreements at regular intervals, whether established for Tech 
Prep or for POS. These intervals ranged from one to three years, though annual meetings 
seemed to be the norm. Postsecondary partners typically hosted these meetings, where in- 
structors and administrators met to discuss the status of the agreement, curriculum align- 
ment, and course materials and equipment. A district CTE administrator described the 
process: “In September or October of [each] school year, we will have articulation meetings 
that are set by the postsecondary institution. .. . The most important person to go is out 


teacher, who is the expert in each area. ... We'll talk about the articulation, how it stands, 
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[and address] any questions or problems. [We consider] do we need to revise it, or do we 
need to continue on for the next year? And some years there are no revisions, and then 


things will change and we'll do some major revisions.” 


At times, these meetings also resulted in more opportunities for secondary students to earn 
postsecondary credit. For example, a district CTE administrator described several instances 
when local community college representatives “discover[ed] that we do something more here 
than they were even aware of. ... They say, ‘Well, gee, if you’re doing that, we can give you 
credit for this.”” These meetings also served to build and reinforce relationships between 


secondary and postsecondary colleagues in many cases. 


In at least one state, the postsecondary partners were reported to have final say in accepting 
or rejecting secondary curricula. According to a postsecondary administrator, “If [an area 
CTE center] wants to offer a course for dual enrollment, then they demonstrate that their 
course content is equal to the course content that we have. ... Through our curriculum ap- 
proval process, the program manager approves the essential objectives for our course. The 


[college] program manager makes that determination.” 


Courses offered for dual or articulated credit might be taught by secondary instructors who 
met community college criteria, community college instructors, or adjunct community col- 
lege faculty. Dual enrollment or dual credit options for CTE coursework were more com- 
mon at area CTE centers in most of the states, and they were also were offered in 
comprehensive high schools in many LEAs. Most of these courses were taught on secondary 
campuses, either at the high school or area CTE center. Joint assessment, or “credit by ex- 
am” (in which secondary students earn college credits by successfully passing the same final 
exam given in the associated postsecondary course), was another option for providing post- 
secondary college credit to secondary students in at least two states. Joint assessments, how- 
ever, were not used by all districts in these states. IHEs in two states required that secondary 
students take the state’s college entrance exam (Accuplacer or the equivalent) to determine 


their eligibility for dual enrollment. 


Respondents were divided about whether the opportunity for postsecondary credit motivat- 
ed students to enroll in POS. Administrators in several districts indicated that this was more 
of a motivator for parents than for students. Some districts were using POS to market CTE 
to students and parents as an option that could lead to postsecondary education. In one dis- 
trict, students enrolled in POS could earn as many as 36 college credits for participation, but 
the fact that secondary students paid half-price postsecondary tuition for these credits was 


seen as a factor limiting participation. 


Many districts had developed dual credit or dual enrollment options outside of the POS 


framework, including Advanced Placement (AP) and International Baccalaureate (IB) 
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coursework. In fact, these other options could be somewhat of an obstacle to POS enroll- 
ment because students might be forced to choose among POS, AP coursework, or other 
electives such as foreign languages. Some LEA officials reported these choices typically re- 
sulted from increasingly stringent graduation requirements for academic coursework, which 
were seen as impeding districts’ ability to offer CTE courses and students’ ability to enroll in 
those courses that were offered. In one state, CTE courses were cut to offer mote academic 
classes. In another state, electives (in which CTE is included in this state) also were cut in fa- 
vor of academic courses. Some local districts have explored ways to address this challenge, 


including extended time (offering more class periods per day) and creative scheduling. 


At both the state and local levels, some respondents suggested that increased graduation re- 
quirements also can be a major obstacle for area CTE centers operating on a shared-time 
model, with students typically taking academic classes in their home high school and CTE 
courses at these centers. Time spent away from the home high school can affect students’ 
ability to meet graduation requirements. At least one area CTE center in one state had begun 


offering academic courses on-site to reduce this obstacle. 


Pathways to Certificates or Degrees 
Secondary and postsecondary respondents in most communities generally regarded the po- 
tential for POS to lead to certificates or degrees as a worthy goal, but this rarely was de- 
scribed as a major focus of POS development and implementation efforts. Secondary 
respondents seemed to see this mission largely as the responsibility of their postsecondary 
partners, whereas postsecondary partners focused more on forging connections with sec- 
ondary agencies when describing their roles in POS. The primary concern expressed by re- 
spondents was the emphasis placed on getting secondary students ‘nto postsecondary 
education with relatively less attention paid to what happened to students after they got 
there. This may reflect the fact that initial implementation of POS was unfolding at the time 


of the case study visits. 


In several states, state and local secondary administrators questioned the practice of adminis- 
tering assessments to provide industry certifications to secondary POS students. In one state, 
local subgrantees were allowed—but not required—to offer industry certification or licen- 
sure examinations. A state secondary administrator explained, “There are some areas that 
simply don’t have industry certification or licenses.” A postsecondary administrator in the 
same state argued that third-party (e., industry) assessments might not be developmentally 
appropriate for secondary POS students and described getting “emotional about the third- 
party assessment thing. We don’t agree that they’re pedagogically sound to implement. Yet 
[the Department] has made it a guiding principle for implementation of skill attainment.” A 
state administrator in another state expressed a similar sentiment, noting that appropriate in- 
dustry certifications might not be achievable within the scope of the secondary component 
of POS. 
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Roles of Employers and Business Partners in State and Local POS Development 
In the majority of LEAs, the primary role of business and industry partners was membership 
on advisory committees. The degree of engagement of these advisory committees varied 
from superficial to deeply involved, but there was consensus across most LEAs that industry 
was more substantively involved than in the past. Several states maintained industry-specific 
advisory councils at the state level, while almost all local districts reported having advisory 
committees in at least one industry area and typically in more than one. Advisory committees 
reviewed curricula, supplied labor market data, identified the technical and “soft” skills they 
want in new hires, and, at some sites, provided guidance on assessments. An administrator in 
one mid-sized district commented on the importance of these relationships: “If a system 
does not have strong ties to the employer community, then it won’t have valid career and 
technical education programs. You can’t have meaningful POS unless you know where your 
students are going, and you have business and industry coming in and working with the stu- 
dents and helping you to understand where you’re going and steering you in the right direc- 


tion.” 


Some small, rural districts reported having difficulty finding appropriate representatives to 
serve on advisory committees; a small district in one state resolved this issue by contacting 
an alumnus, now a business owner in another state, to serve as a virtual advisor, providing 
guidance online and by phone. The district also had received donated equipment from this 
industry partner. According to the district CTE coordinator, “We’ve got to scratch and wig- 
gle... to offer programs to kids that want it. Just because we’re out in the country, doesn’t 


mean that it’s a spot that doesn’t need a program.” 


Transition from Tech Prep to POS 
Four case study states merged their Tech Prep and basic grant funds, and two maintained 
Tech Prep as a separate funding stream. Administrators in three states decided to merge 
Tech Prep and basic grant funds largely because articulation agreements under Tech Prep 
were not seen as effective in helping students make a transition into postsecondary educa- 
tion. In all of these states, administrators observed that few students who earned articulated 
credits were actually using them, perhaps because of the convoluted processes required of 
them once enrolled in community colleges. “Transportability” of these credits was also an is- 
sue, as credits earned under an articulation agreement with one postsecondary institution 
could not always be transferred to other institutions. In one state, officials were devoting 
considerable effort at the state level to make a transition into a dual enrollment process that 


would allow postsecondary credits to “follow” secondary students more easily. 


In the fourth state, Tech Prep and basic grant funds were merged because state administra- 
tors considered it important to bring components of POS to all students, not just the subset 
served by Tech Prep. As one state administrator explained, “We felt like all of the things that 


were so important throughout the career pathways programs of study initiative, that seamless 
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system ... were important for all students, and so we really did not want to just have that 


subset; we wanted to make all of that available and have the districts [be] responsible.” 


Nevertheless, articulation agreements originally established in these states under Tech Prep 
were maintained in some districts even after Tech Prep funds had ceased to be a separate 
funding stream. In at least one local site in one state, local funds were being used to continue 
Tech Prep programming that was no longer supported by Per&zns funds. In some districts, 
POS were seen as virtually indistinguishable from Tech Prep. In many other districts, POS 
were seen as building upon and continuing the work of Tech Prep, and some local district 
and postsecondary staff continued to use the language of Tech Prep. Local district adminis- 
trators in another state lamented the loss of Tech Prep coordinators. One local CTE admin- 
istrator said that “rigorous inclusion of academic quality with technical content” 
distinguished POS from Tech Prep, while others cited the POS emphasis on industry cre- 


dentials as an important distinguishing feature. 


Very different attitudes toward Tech Prep prevailed in the two states that had maintained 
separate funding. In one state, secondary and postsecondary state agencies disagreed on 
whether to merge Tech Prep with the basic grant, but the separate funding stream was main- 
tained after a shared advisory committee for the two bodies reached an impasse in successive 
rounds of voting. The secondary agency eventually agreed to the postsecondary agency’s ar- 
gument for keeping Tech Prep because the alternative was seen as “politically unpalatable,” 
according to the CTE director at the secondary state agency. Secondary staff described hav- 
ing little to show for Tech Prep efforts and were concerned that continuing Tech Prep 
amounted to “dumping money into a black hole.” Postsecondary staff, however, saw Tech 
Prep as having built a strong and continuing foundation for POS. As an administrator at the 
postsecondary agency explained, work now conducted under POS involved merely “popping 
the hood” and updating articulation agreements already established with Tech Prep rather 
than starting from scratch with POS. Another state postsecondary administrator described 


one rationale for maintaining the Tech Prep funding stream: 


The reason we argued to keep it and won that battle was gender equity. 
When [funding for] gender equity [coordinators] went away, so did gender 
equity. When there was no person on the campus responsible for that activ- 
ity, it went away. We didn’t want the same thing to happen with the articu- 
lation agreements that were being updated annually, the connections being 
made between teachers and faculty at the community colleges, and the 
alionment of CTE curriculum. If we don’t fund a Tech Prep coordinator on 
a campus, that activity will go away, because there are no funds to support 


it. 
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In the other state that maintained separate Tech Prep funds, there was consensus across 
state agencies about the strength of the Tech Prep program, and Tech Prep directors played 
a key role in facilitating the development and implementation of POS. In local districts 
across the state, Tech Prep directors facilitated communication and partnerships between 
secondary and postsecondary levels because they had existing “natural links.” One local dis- 
trict specifically used the term “Tech Prep program of study” to refer to programs devel- 
oped with Tech Prep directors facilitating them. In this state, districts that already had well- 
developed Tech Prep programs were described as more likely than others to have well- 
developed POS. 


POS Availability 
According to state administrators, local applicants in all six states were required to offer one 
ot more POS to receive Perkins funding. In some states, the minimum, in practice, was typi- 
cally two or three POS. It was apparent, however, that understanding of and commitment to 


POS varied significantly across local subgrantees. 


Both state and local respondents spoke at length about their plans for expanding POS offer- 
ings in the immediate and long-term future. Plans for expansion were guided by a variety of 
factors: responsiveness to industry, including both local labor market opportunities and 
emerging state or national trends; responsiveness to popular career choices (such as child 
care, health, business); and responsiveness to the interests of local administrators and faculty 


members. Of course, these factors sometimes overlapped. 


Administrators in one state focused on developing statewide POS in the career clusters most 
important to the state economy. “We have taken the statewide perspective based upon in- 
dustry certification and developed that,” said one administrator. A second state engaged in a 
series of visioning and planning sessions using economic data to identify which POS to de- 
velop. An administrator in this state described these planning sessions: “What programs of 
study based on projected economic data, based on the postsecondary employment options 
and postsecondary training institutions are best for [this state]? Why should we be investing 
Perkins fands, time, effort, and energy into creating POS that may not be available to stu- 
dents in this state as future employment options? So those [issues] really helped set the con- 


text of what the program of study is going to look like.” 


Local CTE administrators in several LEAs described the development of POS specifically 
attuned to a local or regional business or industry. For example, local administrators in two 
states noted the creation of a logistics POS in response to the development of a shipping 
port in one case and to a major trucking nexus in another. Another district developed a hos- 
pitality and tourism POS in response to the building of a major new resort that will need 


hundreds of employees with a variety of skills. 
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In other discussions at the local level, it was less obvious that the expansion plans 
were based on an assessment of industry needs. New POS appeared to have been 
selected because they were focused on popular careers or because they reflected the 
interests of individual administrators or faculty members. Comments on POS selec- 
tion included “TI kept hearing a lot about biotechnology”; “I saw more need in health 


sciences”; and “Marketing and tourism are big.” 


In some areas, POS expansion was connected with the policy and vision that college and ca- 
reer readiness are desirable outcomes for all students graduating from the K—12 system. One 
local administrator observed how valuable it would be if CTE teachers used POS more ex- 
tensively as part of their career planning units, if school counseling staff used POS to advise 
students on career paths, and if administrators used the POS concept to organize their think- 
ing about the importance of both career and college for students. A state CTE director not- 
ed that “part of the message that we bring is the fact that this is about college and career 


readiness for all students.” 


POS Quality 
Although state staff reported that one or more POS were in place for each local subgrantee, 
the quality of these POS varied considerably. The CTE director in one state described a con- 
tinuing conversation at the state level about how to identify high-quality POS: “As these 
[POS] come into our [state agency], what criteria are we using other than that they have the 
[four] components? What would we expect to see in a high-quality POS? We want to know 


what ‘good enough’ is and what other levels of ‘good’ are.” 


State Assessments of POS Quality 
Representatives of three states said that they did not yet have any assessment or evaluation 
process for measuring POS quality. Administrators in two of these states said that compli- 
ance with basic requirements is probably the norm, with some exceeding that level. One 
state CTE director said, “My gut tells me that we’re at the compliance level, with some peo- 
ple beyond that. But most people are probably at the compliance level.” These states docu- 
mented only the presence or absence of POS. The third state did not yet have any POS 
beyond the draft stage of development. 


Staff from the other three states concluded that their POS are of good quality based on the 
monitoring systems currently in operation. Two of these states emphasized the effort and at- 
tention that state staff had given to technical assistance and standard setting for POS. As one 
state CTE director observed, “Our staff has gone out and done regional workshops where 
they’ve brought people in, sat them down, and said, ‘Now, understand, these are the expec- 
tations.’ We’ve put a lot of time and energy into that type of technical assistance. So we feel 
pretty good about where we are.” A representative from another state explained, “We’re 


working hard not only to boost rigor within the program standards of our CTE offerings, 
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but then also to recommend really rigorous complementary academic [coursework] for that 


student in that career pathway.” 


Several states had incorporated POS review into consolidated program reviews that included 
other non-CTE programs at the secondary and postsecondary levels. In one state, consoli- 
dated program reviews were conducted on a cycle by the state secondary education agency 
and the state board for community and technical colleges for secondary and postsecondary 
entities, respectively. Administrators in one state described a consolidated program review of 
all federal programs in all school districts on a four-year cycle. Respondents in another state 
department of education, however, noted that monitoring visits have been reduced dramati- 
cally due to budget cuts, leaving them with no way to enforce quality control other than 
reading POS descriptions in the local Perkins plans. Several equated the quality of POS with 
evidence of outcomes such as transitions into postsecondary education or into employment, 


but none provided such evidence. 


Local Assessments of POS Quality 
A majority of local administrators, faculty, and staff offered positive assessments of the qual- 
ity of POS they were developing or implementing and offered a range of explanations for 
how they determined quality. Several used outcomes to judge the quality of POS. As one 
community college director of workforce development stated, “I assess the quality based 
upon the number of students that we know are coming into postsecondary education. So the 
tremendous success is that we know students have started in secondary and are here, and we 
can tell you some of those students then are in bachelor’s [degree] programs with our local 
university.” Other criteria for assessing POS quality offered by LEA and IHE representa- 
tives in several states included rigorous curriculum; technology; very high expectations for 
students; quality of instruction, equipment, and students; active student organizations; and 


student success in competitions. 


Local respondents also noted that administrative buy-in to POS was crucial in promoting 
POS quality. For example, one CTE director noted his engagement in curriculum mapping 
of content areas to reflect the recent revision of state CTE frameworks and to ensure that 
everything required by the frameworks and the industry certifications is included. Another 
administrator noted the willingness of secondary administrators to speak to community col- 
lege administrators about poor instruction in dual credit courses at the college. Another col- 
lege administrator discussed an effort to meet the state objective of improving academics in 


CTE by strengthening the mathematical requirements of an instrumentation program. 


Efforts to Improve POS Quality 
Some state and local officials had moved beyond assessing the quality of POS offerings to 
strengthening these offerings in terms of POS criteria. Administrators in one state decided to 


seek opportunities to link technical assessments with industry certifications rather than at- 
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tempt to create state technical assessments for every program. They also wanted to find na- 
tionally recognized assessments. According to a senior state official, “We decided we don’t 
know enough about it to start big, so let’s start small. We’ve been bringing together focus 
groups in specific areas and having conversations with the faculty and with industry and talk- 
ing about what truly is happening.” A local CTE director in a different state observed, “Per- 
kins lit the fire under us for doing more industry certifications. When we complete that 
program of study, and we hand it to a student, we want to be able to say, ‘And here’s the in- 
dustry certification you can get.” 

Another approach to strengthening POS was to eliminate programs that did not meet the 
criteria at a high level of quality. For example, one state CTE director observed, “It’s making 
sure we have the right programs of study for the kinds of workforce economic development 
opportunities that are here in the state or that regionally we can support.” A high school 
principal in another state said that their decisions on whether to close CTE programs would 
depend on such factors as enrollment, placement, and whether the POS met the high-wage, 


high-demand, high-skills criteria. 


Representatives from three states reflected on their efforts to improve the quality of POS. 
One state CTE director noted that attempts to strengthen POS focused on alignment both 
with postsecondary education and with academic courses and indicators of student success. 
This state also had established a detailed process for improving POS by engaging industry. A 
state agency official added that state staff determined where the weaknesses were and target- 
ed technical assistance accordingly. The CTE director for a second state noted that the state 
is improving POS quality by engaging with teachers to improve the quality and consistency 
of classroom instruction. This state’s CTE director explained, “We’re in the second year of a 
pilot to help teachers to more knowingly do instruction: [design] assignments [and identify] 
quality indicators [of student performance].” In a fourth state, the CTE director observed 


that it was not so much the POS that needed improvement as their local implementation. 


Factors Facilitating POS Development and Implementation 
State and local respondents offered insights into the circumstances that promoted POS im- 
plementation and development in their states. States tended to attribute the existence of 
Tech Prep programs and structures with supporting POS, while local providers often credit- 
ed knowledgeable and committed faculty and staff with driving successful POS implementa- 


tion. 


State Perspectives 
State administrators identified an array of factors that had facilitated POS development and 
implementation. The most common response across states was that prior work in Tech Prep 
and the continued presence of Tech Prep directors and staff strengthened POS development 


and implementation. Officials in three states credited “foundational work with Tech Prep” 
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as a vety important factor in POS development. As one state department of education offi- 
cial observed, “For a lot of the school districts with strong Tech Prep consortiums, [program 
of study] is a natural and an easy fit.” An official in another state noted the state’s history of 
establishing nationally recognized consortia. In these three states, people who had once been 
engaged in Tech Prep also took leadership roles in POS development. As one state repre- 
sentative explained, “We’ve had the luxury of having good local people who know how to 
do Tech Prep. They know how to develop local articulation agreements.” 


State Perspectives on Facilitating Factors for POS 
State staff cited three factors that facilitated POS development and implementation: 


e Initiative by district CTE directors: According to a senior state department of education 
official, many school districts “see the benefit of program[s] of study now. A lot are 
venturing out. They're being very innovative with their CTE programs, and they're 
looking for colleges that will articulate some of the programs with them.” 


e Initiative by postsecondary institutions: In another state, a state official described ex- 
amples of IHEs choosing to pursue POS development in an industry sector in which 
they knew several high schools already had a sequence of courses. 


e Funding flexibility: One state provided an extra year of funding to local subgrantees to 
help in the transition from Tech Prep to POS. The state placed no restrictions on how 
much of recent grant funding was used to meet the state requirements for POS im- 
plementation as long as local subgrantees demonstrated that they were acting in 
partnership with either a secondary or postsecondary provider and meeting state cri- 
teria. 


CTE directors in two states identified provisions of Perkins IV as crucial in promoting POS 
development. One noted that the Perkins requirements “gave us the anchor for this work.” 
Another credited the specific Perkins requirement for aligning CTE and academic education 
as having “made it easier to sell programs of study, because career tech teachers feel like 
they’re at the table [and] they’re accepted more, because they’re part of this whole process. 
Suddenly we’re all on the same chart.” 


In another state, state policy shifts in response to evidence of the effectiveness of CTE facili- 
tated POS development and implementation. In this state, the discovery that the graduation 
rate for CTE concentrators exceeded that for the general population by 10 percent led the 
state board to decide to adopt the three-unit sequence of study for everyone. This gave CTE 
programs a considerable boost, because they had just implemented POS and academic pro- 


grams had no experience with this structure. 
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Local Perspectives 
Knowledgeable CTE teachers and cooperation across education levels were often cited as 
advantages in POS development and implementation. Several local administrators identified 
the critical roles that experienced CTE teachers assumed. One superintendent said, “They’re 
the ones that drive it, really—they’re the ones that want to make sure they’re aligned—that 
their curriculum is aligned, that their kids are prepared.” Another Perkins coordinator at- 
tributed progress in POS development to teachers with industry knowledge, personal con- 
nections at colleges, and a strong interest in POS. Cooperation between secondary and 
postsecondary levels and respect for the contribution each can make to the preparation of 
students was another influential factor. One local director observed, “Having those meet- 


ings, where we sat together and worked the [POS design] out, helped us locally.” 


Barriers to POS Development and Implementation 
State and local secondary and postsecondary respondents identified barriers to POS imple- 
mentation and development, most of which were embedded in the institutional structures 
and organizational culture of the entities charged with developing POS. Barriers also reflect- 
ed the fact that POS, by definition, attempt to cross such divides as secondary- 
postsecondary boundaries and to change habitual patterns, such as the traditional separation 


of CTE and academic studies. 


State Perspectives 
State officials cited several barriers in developing POS. Officials in five states said remote, 
rural school districts face particular challenges in POS development because many do not 
have easy access to an IHE. As one state CTE director noted, “There are remote school dis- 
tricts where [students] have to travel 100 miles or 50 miles to a community college. ... A lot 
of our small school districts ... offer exploratory programs and not necessarily preparatory 
programs. And so for the students to actually achieve [in] preparatory CTE programs, they 
would have to venture out to either a community college that could be 50 miles or more 


[away], or a neighboring school district, which could be 50 miles or more [away].” 


Related issues for rural school districts included having few major businesses with which to 
partner and having too few qualified CTE teachers. One state made a special effort to pro- 
vide technical assistance to struggling rural districts that requested state assistance, assigning 
staff to assist each district in examining its current programs and workforce needs and creat- 
ing three POS per rural district. These efforts included helping to establish an active business 


partnership in each district. 


State officials in five states cited their status as “local control” states as limiting their ability 
to guide local education policy and practice. In one state, local control is a philosophy that 
pervades the education system and is supported by the current governor, as well as some 


state department of education staff. As one senior official noted, “What you see is a lot of 
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encouragement to go in the right direction, rather than mandates. ... The one thing I really 
like about having Perkins money is that with a new plan, we have decided, because we are a 


local control state, to use Perkins as the leverage.” 


The CTE director in a second state described the impact of working this way: “What that 
basically means is that they take a lot of pride and responsibility at the local level for what’s 
going on. We don’t mandate textbooks; we don’t mandate a lot... . And so what we did was 
took a look at what Perkins IV was giving us the opportunity to do, and then we looked at 


how we might provide the leadership to raise the bar of career technical education.” 


In a third state, one [HE leader identified support for local control throughout the state and 
the plethora of local school boards in small districts as factors that make statewide con- 
sistency in POS more difficult. That state also had met resistance from some local subgrant- 


ees on Perkins IV, officials in the state noted. 


Most states identified one or more barriers relating to cooperation and coordination between 
secondary and postsecondary education in POS development and implementation. These in- 
cluded mismatches in organizational structures as well as attitudes and misunderstandings 
that discouraged cooperation. For example, a barrier to POS development in one state was 
the requirement that community college teachers providing instruction to high school stu- 
dents have a secondary teaching certificate; a recent act of the legislature seemed to have 
remedied this situation. In another state, a few IHEs resisted meeting the minimum fe- 
quirement of offering one or more POS, which was a significant issue in places where as 
many as 15 high schools fed into one college. A secondary state administrator described the 
local postsecondary attitude in these cases: “We don’t have to serve the other 14 schools; we 
have our one.” The community college system provided some assistance with this issue, and 
other community colleges in this state were very willing to work with multiple school dis- 


tricts on POS. 


According to one state official, other barriers were the time and effort required to walk fac- 
ulty through the steps and challenges of aligning secondary and postsecondary curriculum; 
arranging the nuts-and-bolts sessions on the step-by-step process of developing articulation 
agreements; and dispelling “those preconceived notions [about secondary students] that exist 


sometimes on the part of the postsecondary faculty.” 


Another barrier cited in some states was the lingering perception that “vocational education” 
was a place to send “problematic” students or those who were not considered “college mate- 
rial.” One state CTE director reported contracting with an outside agency to conduct re- 
search and create a communications plan. Based on the consultants’ recommendations, 


several initiatives were implemented to change public attitudes and perceptions of CTE. 
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These included changing the name from vocational education to CTE, which took an act of 


the state general assembly. 


State officials identified a number of other barriers to POS. These barriers included lack of 
guidance from the Department; pervasive budget issues that led CTE educators to worry 
more about program survival than POS; and lack of knowledge among school counselors 
and scarcity of materials for educating the public and students about POS, sequential courses 


of study, and similar CTE efforts. 


Local Perspectives 


Local district officials identified many of the same barriers as state officials did, including the 
challenges facing rural schools, resistance to federal efforts based on a philosophy of local 


control, and lack of coordination between secondary and postsecondary education. 


Local secondary CTE administrators 


shareda nambevok altieaities dies bail Local Perspectives on POS Barriers 


encountered in working with postsec- Local respondents described specific barriers that might be 


based on attitudes to POS and logistical challenges to col- 
laboration between LEAs and IHEs: 


ondary partners. They said that IHEs 


did not see the benefit of articulation 
agreements; that IHEs were more in- 
terested in the money than in the bene- 
fits to students; and that [HE faculty 
were ambivalent about working with 
secondary educators. IHE administra- 
tors offered parallel concerns about 
working with secondary educators, such 
as maintaining standards in the high 
school class and ensuring that the soft- 


ware used, for example, was current. 


Local districts identified another set of 
barriers to POS development that re- 
flected the circumstances of small rural 
school districts. These included trans- 
portation among widely dispersed 
schools and colleges; lack of staff to de- 


liver POS courses; and a poor “fit” be- 


Postsecondary educators with negative attitudes 
toward dual credit courses offered by secondary 
schools or about the readiness of secondary stu- 
dents for dual credit or enrollment opportunities in 
several communities; 


In a few communities, students were required to 
pay tuition to receive community college credit for 
dual credit coursework; 


In many communities, program and course sched- 
uling challenges made it difficult for students to 
study at two different education institutions at the 
same time; 


Placement exams for IHEs that secondary students 
must pass to take a dual enrollment course served 
as a barrier in local communities in at least two 
states; and 


Curriculum differences between secondary and 
postsecondary programs in the same content area. 


tween POS and small schools (i.e., requirements that might make it impossible for a school 


with one teacher in a CTE area to provide POS). 
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Local administrators also noted the complexities posed by coordination and tracking for a 
college with a mandated service area that includes many high schools and multiple college 


campuses serving thousands of students. 


State and Local Capacity for Collecting Data on POS Enrollment 


The six states and 18 local districts relied on a patchwork of data systems and practices to 
track CTE students. Perkins IV neither offers a statutory definition of a POS student nor 
holds state agencies or local subgrantees accountable for reporting on the number of stu- 
dents participating or their outcomes. As a result, relatively few state or local Perkins IV sub- 
grantees are able to produce data on student involvement in these programs. Similarly, few 
of these systems were equipped to track students from secondary to postsecondary educa- 
tion in a systematic way. A few local districts tracked their students through informal net- 
works and telephone surveys, but none focused on students in POS. One state had 
implemented a longitudinal system for tracking K—12 students into postsecondary education 
and beyond but nonetheless lacked a systematic process specifically for tracking POS stu- 
dents, primarily because this reporting is not required. In some states, administrators were 
planning or taking steps to implement longitudinal data systems, and some states had a few 


elements of such systems in place. 


Current Procedures for Identifying POS Students 
Of the six state departments of education, five reported having some system in place that 
could identify secondary CTE students. In the sixth state, a stand-alone agency (not the state 
department of education) operated a data system that could identify some secondary CTE 
students; the system, however, lacked unique student identifiers and local subgrantees, with 
the exception of those in Tech Prep consortia, were not required to submit data. Most state 
and local respondents could not distinguish POS students from other overlapping categories 
like CTE students, dual enrollment participants, or Tech Prep students. One senior state 
administrator explained the lack of capacity to identify POS students in the state’s data sys- 


tem: 


[Tech Prep and POS are] two different systems. ... Our districts identify 
their Tech Prep students through that articulation agreement with the col- 
leges and credits received at the college. But our program of study is just a 
program of study that school districts must have to comply with Perkins IV. 
And [we do not collect]ny additional information that we would submit for 
further study to see if a student was actually in this specific program of 
study [or] went to the college. We don’t have that kind of a longitudinal da- 


ta following our students in our program of study. 
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Only one state had a nascent capacity to distinguish students enrolled in the secondary por- 
tion of POS from those enrolled in CTE programs overall due to a relatively new student- 
level data system. This state department of education’s electronic portal, in which all second- 
ary schools and consortia register their POS, was also the point of access for Perkins ac- 
countability data collection. Subgrantees provided data on a template organized by POS 
program codes for the 16 career cluster-based POS developed at the state level. In this sys- 
tem, each student has a unique identifier, enabling disaggregation of the data. Because LEAs 
provided data on every student’s level of participation by program code and student identifi- 
ef, state administrators will soon be able to improve their tracking of POS enrollees. As one 
state administrator explained, “Ultimately once we start collecting course code data, then 
we'll be able to align the database of the programs of study that are approved with the 


courses that are indeed offered and the students [who] take those courses.” 


While one state had a longitudinal data system in place for tracking K—12 students, the sys- 
tem was not set up to track POS students. The CTE director in this state asserted that POS 
need to be tied to performance accountability measures and said that “having some clear 

guidance from the Department in terms of the measurement approach that we use for that 


[purpose] would be very helpful.” 


Less comprehensive systems in other states gathered some data on students’ transition from 
secondary to postsecondary CTE programs. For example, one state had a system designed to 
collect dual credit and dual enrollment data for secondary students. Another state also had 
continuation data for dual enrollment students entering community colleges or the state col- 
lege system. In a third state, the Tech Prep program recently installed a new data system that 
could track students from secondary to postsecondary education. As one administrator not- 
ed, “This is the first time that the two systems are going to be able to talk to each other, and 
so data down the road will be better... because they will then be able to track [Tech Prep 
students] with the student information from high school to college into the courses of 


study.” 


In addition, most states had databases of postsecondary students, but these typically were 
limited to one postsecondary system (1.e., community colleges, but not the state four-year 
college system). They also might not yet have the capacity to distinguish students on a CTE 


track from those on an academic trajectory or to distinguish those participating in POS. 


Tracking Secondary Students into Two-Year or Baccalaureate Institutions 
Few data systems had the capacity to track secondary students into postsecondary institu- 
tions. Those that did were not comprehensive; they could only track a subset of CTE stu- 
dents into a subset of state postsecondary institutions. Two states reported that they could 
track Tech Prep or dual enrollment students from secondary education into the state post- 


secondaty system; in one case, this included the ability to track dual enrollment students into 
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both the community college system and the state four-year college system. None of these 
systems was yet capable of distinguishing students engaged in POS from those engaged in 
CTE or Tech Prep programs. As one state official from a state with substantial data- 

collection capacity observed, “The answer is we currently don’t have an easy way at either 


system office to watch students move from one segment to another.” 


One state operated a longitudinal data system that tracked secondary CTE students into 
postsecondary institutions. The system also tracked student occupational competencies out- 
lined in the state curriculum frameworks and tied them to a standard classification code. 
Many students could not be tracked, however, because not all state colleges—including the 


state university system—participated in the system. 


In at least three states, administrators reported participating in the National Student Clear- 
inghouse (NSC) to track students in postsecondary education. One state purchased a 
“statewide license,” which gave LEA staff free access to the NSC and enabled state staff to 
locate some CTE concentrators who had moved into postsecondary education through a da- 
ta match. According to the staff person in charge of this process, “We’re in our second year 
of doing a data match, where we’re able to take the big file of all the concentrators from [the 
state’s] data warehouse, send it up to the Clearinghouse, and then get back a report that says, 
‘Of those kids that you sent up, these are the ones that we found. Here are the ones that are 
full-time, half-time, and [here are the] institutions they’re going to, which ones they trans- 
ferred to.”” The same staff person described the results of this data matching: “The first time 
that we did it, we had basically a 72 percent college-going rate for our concentrators com- 


pared to a 68 percent overall college-going rate for the state in general.” 


Plans for Developing Tracking Systems 
Most of the plans for new data systems apparently were motivated by the desire for the state 
to have the capacity to track CTE students after they leave secondary education. One state’s 
next steps in the development of its data system included plans to create an “automated 
CTE high school graduate follow-up system.” Three states were seeking substantial funding 
for longitudinal systems: one state had plans for a longitudinal data system, for which they 
were hoping to receive federal Race to the Top support; another state was pursuing a 
Statewide Longitudinal Data System (SLDS) grant for the same purpose; and a third state 
had submitted a proposal for funds to establish a longitudinal data system across all the lev- 
els of postsecondary and K-12 education as well as for data matching with the employment 


development department and Unemployment Insurance (UI) wage data. 


In one state where the existing secondary database was operated by a stand-alone agency, the 
state department of education was also establishing a new database that would improve its 


capacity to collect student-level data. The database was intended also to address the quality 
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of the CTE data. One state administrator explained how the database under development 


would work: 


We [will] have identifiers at the district, school, course, section, and student 
levels. With those, we can have discrete measures relative to CTE, or we 
can put attributes on things. For an agricultural biology course, for example, 
they can list it and code it as a biology course, but the district can indicate 
that it’s also a CTE course. What we were talking about doing .. . once this 
huge data reservoir is there, [is dipping] our straw into that. If we have a 
computerized application that describes what the school or the district’s 
POS is, we should be able to electronically match up their application with 


what actually shows up in the data. 


Barriers to Collecting Data on POS Enrollment 
State officials identified several barriers to tracking POS student outcomes. Three states re- 
ported that challenges related to the Family Educational Rights and Privacy Act (FERPA) or the 
use of student Social Security numbers impeded their efforts to establish new databases that 


bridge secondary and postsecondary education. 


Officials in two states bluntly referred to the concept of tracking POS outcomes as “an un- 
funded mandate.” As one state official explained, “Had that been built into Perkins IV legis- 
lation, then we would’ve looked at it differently, definitely, of course. But it wasn’t, and so 
we just followed the language of the law.” The second official observed, “All of us [states] 
will have to be buying degree audit software that will allow us to classify students into multi- 
ple POS.” 


In addition, officials in one state questioned whether the data would be useful, even if they 
could be gathered. They noted that data on the number of students who enrolled in and 
completed POS cannot reveal the success of the POS because students who participated in 
POS but determined that a particular career pathway was not for them may nonetheless have 
valuable outcomes. Even if the goal is simply to measure how CTE students are progressing 
along the educational continuum, POS would be too limited a category. Officials from sev- 
eral local districts made similar observations—in other words, that students who leave a pro- 
gram early to obtain employment, or students who attend multiple colleges to complete their 


training, will not be “counted the way they should be counted.” 
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Summary 


The study team conducted case study visits in six states and three local communities within 
each state during the second year of Perkins IV implementation. Interviews with more than 
130 respondents revealed that most ot these state and local practitioners were making seti- 
ous efforts to develop and implement POS, establish more sophisticated accountability sys- 
tems, and stretch limited Perkins funds to comply with statutory expenditure requirements 
and limitations (e.g., caps on administrative and leadership funds). Although the interviews 
occurred relatively early in the life cycle of Perkins IV, they revealed a number of creative ap- 
proaches at the state and local levels. State and local administrators described steps taken to 
move to web-based reporting systems for collection and compilation of Perkins accountabil- 
ity data, strategies for engaging secondary and postsecondary administrators and faculty in 
the creation of POS that respond to labor market needs, and efforts to use limited Perkins 
funds to strengthen CTE programs and enhance coordination—both between academic and 


CTE instruction and across the secondary and postsecondary levels. 


Variations across and within states provided dramatic evidence of the flexibility inherent 
within Perkins IV. While all respondents described efforts to comply with the same statutory 
provisions, the case studies revealed significant differences in interpretation and a range of 
adaptations based on state and local context, needs, resources, and priorities. Although some 
implementation strategies were reportedly working better than others, no state, LEA, or IHE 
could be identified as having all of the statutory elements in place and as working effectively 
for all populations. Most respondents were quick to point to their accomplishments with re- 
gard to POS, accountability systems, and uses of Perkins funds—and equally willing to 
acknowledge the limitations or shortcomings of strategies attempted in their states or com- 


munities. 


Nonetheless, a number of common themes, strategies, concerns, and frustrations were iden- 
tified at all or most of the case study sites. These findings may not be representative of all 
states, LEAs, and IHEs, but they provide a detailed exploration of the status of Perkins tm- 
plementation in six states specifically chosen for location, size, and demographic diversity 
and in urban, suburban, and rural communities within these states. The next three sections 
provide a recap of key findings in the three main topic ateas: Perkins IV expenditures; ac- 


countability systems; and POS. 


Finance Systems 


Despite reports of serious budget constraints, all of the case study states and most of the 
LEAs and IHEs provided funding for CTE through categorical grants or general budget ex- 
penditures for teachers, administrators, facilities, and classroom materials. Nonetheless, al- 


most all respondents indicated that the need for Perkins funds exceeded the dollars available. 
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State CTE administrators reported that administrative funds were inadequate, especially with 
the new requirements of Perkins IV. They also noted that the amount designated for leader- 
ship funds was insufficient to cover expanded categories of required and permissible uses. 
Similarly, LEA and IHE administrators agreed that the amount of Perkins funds they re- 


ceived was insufficient to meet needs and statutory requirements. 


Across the states, decisions about the secondary and postsecondary funding split were based 
on factors such as data analyses, competitive grant processes, the number of eligible provid- 
ers, and comparative enrollments. Four of the six case study states merged Tech Prep funds 
with their basic Perkins grants, and four used the reserve option to enhance flexibility, in- 
crease funding for rural communities, and support innovation and technical assistance. In 
most of the states, a sizeable number of LEAs and IHEs formed consortia to meet the min- 


imum allocation threshold. 


Accountability Systems 


Almost all state and local CTE administrators interviewed recognized the need for a Perkins 
accountability system, but many questioned whether some of the core indicators are appro- 
priate measures of performance (specifically, the indicators for academic attainment and 
nontraditional participation and completion) and cited numerous barriers to collecting the 
required data. Nonetheless, most state and local CTE administrators and faculty appeared to 
be taking the new reporting requirements seriously and placing greater emphasis on collect- 
ing and reporting valid and reliable data. The sophistication of the Perkins accountability sys- 
tems varied significantly across (and sometimes within) the case study states. State and local 
administrators identified shortcomings in all of the accountability systems, with none report- 
ing the capacity to track CTE students systematically from secondary to postsecondary edu- 
cation. Although state CTE administrators generally exhibited a thorough understanding of 
Perkins tepotting requirements and a commitment to helping LEAs and IHEs provide valid 
and reliable data, local case study visits suggested significant differences in their understand- 


ing of the state’s performance measures and the populations to be reported. 


State CTE administrators generally appreciated the nonregulatory guidance provided by the 
Department, but they liked the flexibility to create their own definitions and performance 
measures based on state data sources, reporting capacity, and program delivery structures. 
All of the case study states used at least one of the Department’s recommendations for 


measurement populations and performance measures, but none adopted all of them. 


Reporting capacity within the states ranged from sophisticated longitudinal systems to more 
rudimentary systems that required local staff to compile or “cut and paste” data from other 
sources. A number of administrators noted that the cost of Perkins data collection and re- 


porting far exceeds the human and financial resources available for this purpose, and most 
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cited the need to identify other funding sources to support the development of sophisticated 
accountability systems. State administrators often reported using local Perkins performance 
data to identify programs in need of improvement, but local administrators offered varied 
opinions about the extent to which local performance targets actually led to improved per- 
formance. Similarly, state administrators typically expressed confidence in their ability to re- 
port disaggregated data on most of the special populations identified in Perkins IV, but local 
CTE practitioners were less sanguine about their capacity to provide these data and more 


explicit in their descriptions of the challenges associated with collecting them. 


Programs of Study 


State CTE administrators in case study states indicated that they incorporated the four statu- 
tory POS components into technical assistance efforts and approval criteria used for local 
POS, but interpretation of the four components was not always consistent across or within 
states. Although most state administrators expressed opposition to regulatory language de- 
fining POS, a number said that the Department’s POS Design Framework would have been 
helpful earlier in the process of POS development and implementation. At both the second- 
ary and postsecondary levels, perceptions of POS varied considerably: some CTE officials 
identified the POS concept as the most important change in Perkins IV’, while others viewed 


it as simply a new name for an existing approach. 
ply ig app 


Processes and policies for POS development, approval, and implementation varied across 
and within states—and most CTE administrators advocated for continued flexibility in these 
areas. Development of effective partnerships between secondary and postsecondary systems 
was a significant hurdle in many sites despite widespread understanding that coordination is 
a major goal of Perkins IV. The relationships and processes established under Tech Prep 
played a prominent role in POS development and implementation in many places. With no 
accountability measures required for POS in Perkins IV, state and local CTE administrators 
in case study sites were rarely able to distinguish POS participants from other CTE students. 


None reported systematic capacity to track POS enrollment or outcomes 


Concluding Thoughts 


Despite variation in approaches to Perkins IV implementation across and within states, the 
case studies identified many commonalities—including a common tesistance to the possibil- 
ity of losing the flexibility afforded by the current legislation. The case studies highlighted 
the trade-offs associated with providing flexibility at the state and local levels while seeking 
consistency in data collection and reporting. Although both flexibility and consistency are 
important goals, the case study findings indicate that it may not be possible to achieve both 
simultaneously and suggest that priorities may need to be determined in future Perkins reau- 


thorizations. 
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Attachment A. Interview Protocols 
State Secondary CTE Director 


Context/Perspective 
1. What are your current roles/responsibilities in the implementation of Perkins IV? 
(Probe on changes since Perkins LI, role related to CTE funding, role related to 
CTE accountability, and role related to programs of study [POS].) 


2. How does CTE fit into your larger state education reform efforts? 


Programs of Study 
PROGRAMS OF STUDY CREATION: This section will help us to understand how POS 


were developed within the state. See the relevant state survey questions for background. 


3. What progress has the state made toward developing POS? What is the working 
definition of a POS in your state and how was it developed? How was it communi- 


cated to local education agencies (LEAs) and postsecondary institutions? 


4. What is the approval process for POS in the state? (Probe on approval criteria and 
obtain a copy if there are written criteria.) Have you rejected any POS submitted to 
the state for approval? (If yes, probe on how many and why. If no, probe on rea- 
sons for which a POS might be rejected.) 


5. In your state survey you indicated that your state agency played the following roles 
in developing POS (list from state survey). Can you provide us with additional de- 


tails or clarifying information on POS development? 


6. How would you characterize the typical process used to develop POS within local 
communities? For example, was POS development at the local level usually led by 
school districts, postsecondary institutions, or the two working as partners? How 


did employers and other business groups participate in the development process? 


7. How comparable are POS in similar pathways (e.g., construction or health) that are 
offered by different local providers? Ate common curricula and assessments used 


in similar POS across the state? 


8. Would your state support the development of statutory language and/or federal 
regulations defining the components of POS? If yes, what types of language would 
you support? If no, why not and what type of direction, if any, would you support? 


POS IMPLEMENTATION: These questions will help us understand how implementation 
of POS occurred at both the state and local levels. 
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10. 


11. 


12. 


13. 


What progress has the state made toward implementing POS? Specifically, what 
factors have facilitated implementation? What barriers have you encountered? 


(Probe on specific facilitating factors and barriers.) 


Did you have sufficient information to support districts and their postsecondary 
partners in developing POS that include the statutory elements? (Probe on what in- 
formation was available and what wasn’t.) What types of additional information or 
support from the Office of Vocational and Adult Education (OVAE) would have 
been helpful? 


In your state director survey, you indicated that your state provides the following 
types of technical assistance (review list) related to implementation of POS. Are 

there any other types of support that were offered? To what extent have districts 
and postsecondary institutions participated in these technical assistance activities? 


Which ones wete most successful? Least successful? 


Is at least one POS currently available in each district or consortia? (Probe on why 
ot why not.) What percentage of local providers is offering more than one POS? 


What is the state’s plan for expanding POS offerings in the future? 


What is your assessment of the general quality of the POS offered in the state? 
(Probe on the characteristics of high-quality POS and the steps that are being taken 


to improve quality if there are concerns in this area.) 


ASSESSING OUTCOMES OF POS IMPLEMENTATION: Although states are not re- 


quired to report on POS outcomes, these questions will help us understand how states are 


measuring the effects of these programs. 


14, 


15. 


Do you have a statewide system or process for tracking POS enrollment? For 
tracking POS outcomes? (If yes, who maintains it? Is it differentiated by program? 
Does it contain data on both secondary and postsecondary students?) If the state 


does not have a tracking system, are there plans to develop one? 


Do you have the capacity to track or follow secondary POS students into two-year 
postsecondary institutions in the state? Into baccalaureate institutions? Is this ca- 
pacity limited to secondary students who remain in the same POS from secondary 
to postsecondary or those who remain within an established second- 

aty / postsecondary partnership? (Probe on extent of state’s capacity to track sec- 
ondary to postsecondary transitions and ask for summaries or examples of tracking 
data that show extent to which students who begin a POS at secondary level con- 


tinue at postsecondary level.) 
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Funding 


DISTRIBUTION OF FUNDING: These questions will help us understand how the exist- 
ing allocation formula supports or inhibits POS development in particular and the attain- 


ment of the Act’s provisions in general. 


16. 


ae 


18. 


19. 


20. 


21; 


22. 


23: 


24. 


In your state plan update, you indicated that your state is allocating the following 
percentages of funding to the secondary and postsecondary sectors: [Provide in- 
formation]. What are the reasons for this split? (Probe on how determined and 
changes from Perkins III.) If applicable, how does your state allocate Perkins IV dol- 
lars to area or regional vocational/technical schools? (Probe on changes since Per- 
kins III.) 


Your state plan indicated that you ARE or ARE NOT using the reserve fund. 
Please tell us a little more about the reasons for this decision. (If state is using the 


reserve fund, probe on how the funds are being targeted.) 


What must local education agencies (LEAs) do to obtain Perkins funding? (Probe 
on application process, use of RFPs, use of local plans, and reasons why a district 


might not get funding.) 


Were there any districts or postsecondary institutions in your state that were unable 
to qualify for the minimum allocation and unable to form consortia? If yes, were 


waivers used to provide Perkins funding? (Probe on why or why not.) 


How ate Perkins state leadership funds used? (Probe on changes since Perkins III 
and whether leadership funds are sufficient to support required and permissible us- 


es, including POS development.) 


How are Perkins IV’ state administrative funds used? (Probe on changes since Per- 
kins III.) How has the state met matching requirements for use of Perkins adminis- 
trative funds? (Probe on whether funding is sufficient to support POS 


development, accountability provisions, and other administrative functions.) 


What suggestions would you offer for changes in funding provisions in future Per- 
kins legislation? (Probe on specific changes to uses of reserve, leadership, and ad- 


ministrative funds.) 


Have expenditures for non-traditional training and employment changed under 
Perkins IV? Why ot why not? (Probe on whether state believes continuation of 


nontraditional measures and programs is warranted. 


You indicated in your state survey that your state (HAS or HAS NOT) merged its 


Tech Prep funding into its basic grant. 
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o Ifyour state DID NOT merge Tech Prep, how are Tech Prep funds dis- 
tributed and why? (Probe on formula vs. competitive and criteria for distri- 


bution. If formula, what is it?) 


o Ifyour state DID merge Tech Prep, what has happened to existing net- 
work of Tech Prep programs and coordinators? (Probe on bene- 


fits /drawbacks of merger.) 
o In either case, how does a Tech Prep program differ from a POS? 


25. Did CTE recetve ARRA funds in your state? (Probe on why or why not and for 


what purposes, if yes.) 


Accountability 
ACCOUNTABILITY SYSTEM DESIGN: These questions will help us understand how 
your state measures Perkins outcomes and how federal nonregulatory guidance has supported 


development of its accountability system. 


26. According to your state five-year plan submission, your state has defined a CTE 
participant as [fill in the blank] and a CTE concentrator as [fill in the blank]. How 
has your state communicated these definitions to LEAs? How do LEAs identify 


these individuals and how consistent are these populations across agencies? 


27. What steps does your state take to support the collection of valid and reliable Per- 
kins IV accountability data from local subgrantees? (Probe for designation of re- 
sponsibilities [1.e., who does what], types of training and TA provided, types of 


follow up to assure data quality, and changes since Perkins III.) 


28. How did the state use the nonregulatory guidance provided by the OVAE to sup- 
port its efforts to construct measures and define measurement populations? (Probe 
on extent of use and why.) Do you believe that OVAE should issue regulations re- 
garding measurement populations, measure construction, and reporting processes 


for Perkins? (Probe on why or why not.) 


29. In our review of your state plan submission, we see that your state has adopted the 
following measure constructions. [Insert measure constructions.] How does your 
state gather information to report on these measures? What challenges are associat- 
ed with data collection? (If the state chose not to folow OVAP’s nonregulatory 
guidance for one ot more measures, ptobe on reasons for selection of different 


measures.) 


DIFFICULTY IN COLLECTING DATA: These questions are intended to assist us in un- 
derstanding the obstacles states face in collecting data on the measures and specific sub- 


populations of students. 
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Other 


30. What is your state’s capacity to report on special populations and subpopulations 
(e.g., individuals with disabilities, single parents including teen parents, displaced 
homemakers, economically disadvantaged students including foster children, aca- 
demically disadvantaged students, English language learners, students pursuing oc- 
cupations that are nontraditional for their gender)? How useful is this information? 


What changes, if any, would you suggest that future legislative efforts address? 


31. How burdensome are the Perkins accountability measures at the state, district, and 
school/institution levels? What recommendations would you suggest to reduce 


burden in future legislation? 


32. What investment has the state made in data systems or staffing to meet Perkins IV 
data requirements? (Probe on changes in staff FTEs and resources required to 
build or adapt management information systems, design surveys, train staff, etc.) 
How will the adoption of state longitudinal data systems affect future reporting? 


Have the new accountability requirements required investments at the local level? 


USE OF DATA TO SUPPORT PROGRAM IMPROVEMENT: These questions will help 


us understand how the state and local agencies are using data. 


33. How are accountability data used by the state? By eligible recipients? 


34. How were adjusted performance levels on core indicators negotiated with OVAE? 


(Probe on suggested changes to the negotiation process.) 


35. How were adjusted performance levels on core indicators negotiated with districts 
and postsecondary institutions? To what extent have these local performance tar- 
gets affected the state’s ability to identify programs in need of improvement? To 
what extent have they contributed to improved performance at the state level? 
Should local performance targets be included in future legislation? (Probe on why 


ot why not.) 


36. What rewards and sanctions are available and used? (Probe on criteria for threshold 
levels for state actions, process used, and impact on local programs.) Have any lo- 
cal programs been subject to the specific sanctions outlined in Perkins IV and, if so, 


have the sanctions led to program improvement? 


37. How is information about CTE communicated to the state superintendent, the 
state board of education, and the general public? (Ask for examples of reports or 


agenda items.) 


38. Does the state award academic credit for coursework completed in a CTE subject 


area or class? Does the state have any plans to introduce or expand this option? 
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39. What are the certification requirements for CTE teachers in the state? Are they 
similar to the certification requirements for other secondary teachers? Is alternative 
licensure an option for CTE teachers? (Probe on challenges associated with finding 
qualified CTE teachers and options for getting them into the classroom.) 


State Postsecondary CTE Director 


Context/Perspective 
40. What are your current roles/responsibilities in the implementation of Perkins IV? 
(Probe on changes since Perkins III, role related to CTE funding, role related to 
CTE accountability, and role related to programs of study [POS].) 


Programs of Study 
PROGRAMS OF STUDY CREATION: This section will help us to understand how POS 


were developed within the state. See the relevant state survey questions for background. 


41. What progress has the state made toward developing POS? What is the working 
definition of POS in your state and how was it developed? How was it communi- 


cated to postsecondary institutions and local educations agencies (LEAs)? 


42. What is the approval process for POS in the state? (Probe on approval criteria and 
obtain a copy if there are written criteria.) Have you rejected any POS submitted to 
the state for approval? (If yes, probe on how many and why. If no, probe on rea- 


sons for which a POS might be rejected.) 


43. In your state survey you indicated that your state agency played the following roles 
in developing POS (lst from state survey). Can you provide us with additional de- 


tails or clarifying information on POS development? 


44. How would you characterize the typical process used to develop POS within local 
communities? For example, was POS development at the local level usually led by 
school districts, postsecondary institutions, or the two working as partners? How 


did employers and other business groups participate on the development process? 


45. How comparable are POS in similar pathways (e.g., construction or health) that are 
offered by different local providers? Are common curricula and assessments used 


in similar POS across the state? 


46. Would your state support the development of statutory language and or federal 
regulations defining the components of POS? If yes, what types of language would 
you support? If no, why not and what type of direction, if any, would you support? 
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47. 


48. 


49. 


50. 


51. 


52. 


53. 


54. 


55. 


POS Implementation: These questions will help us understand how implementa- 


tion of POS occurred at both the state and local levels. 


What progress has the state made toward implementing POS? Specifically, what 
factors have facilitated implementation? What barriers have you encountered? 


(Probe on specific facilitating factors and barriers.) 


Did you have sufficient information to support postsecondary institutions and their 
district partners in developing POS that include the statutory requirements? (Probe 
on what information was available and what wasn’t). What types of additional in- 
formation or support from the Office of Vocational and Adult Education (OVAE) 
would have been helpful? 


In your state director survey, you indicated that your state provides the following 
types of technical assistance (review list) related to implementation of POS. Are 
there any other types of support that were offered? To what extent have postsec- 
ondary institutions and districts participated in these technical assistance activities? 


Which ones wete most successful? Least successful? 


Is at least one POS currently available in each postsecondary institution? (Probe on 
why or why not.) What percentage of postsecondary institutions is offering more 


than one POS? What is the state’s plan for expanding POS offerings in the future? 


What is your assessment of the general quality of the POS offered in the state? 
(Probe on the characteristics of high-quality POS and the steps that are being taken 


to improve quality if there are concerns in this area.) 


Assessing Outcomes of POS Implementation: Although states are not required to 
report on POS outcomes, these questions will help us understand how states are 


measuring the effects of these programs. 


Do you have a statewide system or process for tracking POS enrollment? For 
tracking POS outcomes? (If yes, who maintains it? Is it differentiated by program? 
Does it contain data on both secondary and postsecondary students?) If the state 


does not have a tracking system, are there plans to develop one? 


Do you have the capacity to track or follow secondary POS students into two-year 
postsecondary institutions in your state? Into baccalaureate institutions? Is this ca- 
pacity limited to secondary students who remain in the same POS from secondary 
to postsecondary or those who remain within an established second- 

aty/ postsecondary partnership? (Probe on extent of state’s capacity to track sec- 
ondary to postsecondary transitions and ask for summaries or examples of tracking 
data that show extent to which students who begin a POS at secondary level con- 


tinue at postsecondary level.) 


APPENDIX C. CROSS CASE REPORT 


Funding 


DISTRIBUTION OF FUNDING: These questions will help us understand how the exist- 
ing allocation formula supports or inhibits POS development in particular and the attain- 


ment of the Act’s provisions in general. 


50. 


57. 


58. 


59. 


60. 


61. 


62. 


63. 


64. 


In your state plan update, you indicated that your state is allocating the following 
percentages of funding to the secondary and postsecondary sectors: [Provide in- 
formation]. What are the reasons for this split? (Probe on how determined and 
changes from Perkins III.) If applicable, how does your state allocate Perkins IV dol- 
lars to area or regional vocational/technical schools? (Probe on changes since Per- 
kins III.) 


Were there any districts or postsecondary institutions in your state that were unable 
to qualify for the minimum allocation and unable to form consortia? If yes, were 


waivers used to provide Perkins funding? (Probe on why or why not.) 


What must postsecondary institutions do to obtain Perkins funding? (Probe on ap- 
plication process, use of RFPs, use of local plans, and reasons why an institution 


might not get funding.) 


Your state plan indicated that you ARE or ARE NOT using the reserve fund. 
Please tell us a little more about the reasons for this decision. (If state is using the 


reserve fund, probe on how the funds are being targeted.) 


How ate Perkins state leadership funds used? (Probe on changes since Perkins III 
and whether leadership funds are sufficient to support required and permissible us- 


es, including POS development.) 


How are Perkins IV’ state administrative funds used? (Probe on changes since Per- 
kins III.) How has the state met matching requirements for use of Perkins adminis- 
trative funds? (Probe on whether funding is sufficient to support POS 


development, accountability provisions, and other administrative functions.) 


What suggestions would you offer for changes in funding provisions in future Per- 
kins legislation? (Probe on specific changes to uses of reserve, leadership, and ad- 


ministrative funds.) 


Have expenditures for nontraditional training and employment changed under Per- 
kins IV? Why or why not? (Probe on whether state believes continuation of non- 


traditional measures and programs is warranted.) 


You indicated in your state survey that your state (HAS or HAS NOT) merged its 
Tech Prep funding into the basic grant. 
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o Ifyour state DID NOT merge Tech Prep, how are Tech Prep funds dis- 
tributed and why? (Probe on formula vs. competitive and criteria for distri- 


bution. If formula, what is it?) 


o Ifyour state DID merge Tech Prep, what has happened to existing net- 
work of Tech Prep programs and coordinators? (Probe on bene- 


fits /drawbacks of merger.) 
o In either case, how does a Tech Prep program differ from a POS? 


65. Did CTE receive ARRA funding in your state? (Probe on why or why not and for 


what purposes, if yes.) 


Accountability 


ACCOUNTABILITY SYSTEM DESIGN: These questions will help us understand how 
your state measures Perkins outcomes and how federal nonregulatory guidance has supported 


development of its accountability system. 


66. According to your state five-year plan submission, your state has defined a CTE 
participant as [fill in the blank] and a CTE concentrator as [fill in the blank]. How 
has your state communicated these definitions to postsecondary institutions? How 
do postsecondary institutions identify those individuals and how consistent are 


these populations across institutions? 


67. What steps does your state take to support the collection of valid and reliable Per- 
kins IV accountability data from local subgrantees? (Probe for designation of re- 
sponsibilities [i.e., who does what], types of training and TA provided, types of 


follow up to assure data quality, and changes since Perkins III.) 


68. How did the state use the nonregulatory guidance provided by OVAE to support 
its efforts to construct measures and define measurement populations? (Probe on 
extent of use and why.) Do you believe that OVAE should issue regulations re- 
garding measurement populations, measure construction, and reporting processes 


for Perkins? (Probe on why or why not.) 


69. In our review of your state plan submission, we see that your state has adopted the 
following measure constructions. [Insert measure constructions.] How does your 
state gather information to report on these measures? What challenges are associat- 
ed with data collection? (If the state chose not to follow OVAE’s nonregulatory 
guidance for one ot more measures, ptobe on reasons for selection of different 


measures.) 
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70. Difficulty in Collecting Data: These questions are intended to assist us in under- 
standing the obstacles states face in collecting data on the measures and specific 


subpopulations of students. 


71. What is your state’s capacity to report on special populations and subpopulations 
(e.g., individuals with disabilities, single parents including teen parents, displaced 
homemakers, economically disadvantaged students including foster children, aca- 
demically disadvantaged students, English language learners, and students pursuing 
occupations that are nontraditional for their gender)? How useful is this infor- 
mation? What changes, if any, would you suggest that future legislative efforts ad- 


dress? 


72. How burdensome ate the Perkins accountability measures at the state, district, and 
school/institution levels? What recommendations would you suggest to reduce 


burden in future legislation? 


73. What investment has the state made in data systems or staffing to meet Perkins IV 
data requirements? (Probe on changes in staff FTEs and resources required to 
build or adapt management information systems, design surveys, train staff, etc.) 
How will the adoption of state longitudinal data systems affect future reporting? 


Have the new accountability requirements required investments at the local level? 


Program Improvement 
USE OF DATA TO SUPPORT PROGRAM IMPROVEMENT: These questions will help 


us understand how the state and local agencies are using data. 


74. How are accountability data used by the state? By eligible recipients? 


75. How were adjusted performance levels on core indicators negotiated with OVAE? 


(Probe on suggested changes to the negotiation process.) 


76. How were adjusted performance levels on core indicators negotiated with postsec- 
ondary institutions and districts? To what extent have these local performance tar- 
gets affected the state’s ability to identify programs in need of improvement? To 
what extent have they contributed to improved performance at the state level? 
Should local performance targets be included in future legislation? (Probe on why 


ot why not.) 


77. What rewards and sanctions are available and used? (Probe on criteria for threshold 
levels for state actions, process used, and impact on local programs.) Have any lo- 
cal programs been subject to the specific sanctions outlined in Perkins IV and, if so, 


have the sanctions led to program improvement? 
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Other 


78. 


79. 


How is information about CTE communicated to community college administra- 
tors, governing bodies (e.g., boards of trustees or governors), and the general pub- 


lic? (Ask for examples of reports or agenda items.) 


What are the employment criteria for postsecondary CTE instructors in the state? 
Are they similar to the criteria for other postsecondary instructors? (Probe on chal- 
lenges associated with finding qualified CTE instructors and options for getting 


them into the classroom) 


Local Secondary CTE Director 


Context/Perspective 


80. 


81. 


What are your current roles/responsibilities in the implementation of Perkins IV? 
(Probe on changes since Perkins LIT, role related to career and technical education 
(CTE) funding, role related to CTE accountability, and role related to programs of 
study [POS].) 


How does CTE fit into your district’s broader education reform efforts? 


Programs of Study 
POS CREATION: This section will help us to understand how POS were developed within 


the state. See the relevant state survey questions for background. 


82. 


83. 


84. 


85. 


80. 


What progress has the district made toward developing POS? What is the working 
definition of a POS in your state and how was it developed? How was it communi- 


cated to local education agencies (LEAs)? 


What role(s) does the state agency play in the development of POS? Are POS de- 
veloped at the state level, the district/consortia level, or both? Is there a template 


ot format that the state uses to guide POS development? 


What role(s) does the district play in the development of POS? Is POS develop- 
ment in your community typically led by secondary institutions, postsecondary in- 
stitutions, or the two working as partners? How did employers and other business 


groups participate in the development process? 


How comparable are POS within a given pathway (e.g., construction or health) 
across the state? Are common curricula and assessments used in similar POS 


across the state? 


POS Implementation: These questions will help us understand how implementa- 


tion of POS occurred at both the state and local levels. 
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87. 


88. 


89. 


90. 


91, 


92. 


93. 


94. 


What progress has the district made toward implementing POS? Specifically, what 
factors have facilitated implementation? What barriers have you encountered? 


(Probe on specific facilitating factors and barriers.) 


What kinds of technical assistance does the state provide to districts for implemen- 
tation of POS? (Probe for specific examples such as workshops or individual con- 
sultations.) To what extent did your district participate in these activities? Did you 
have sufficient information and support on POS development and implementa- 


tion? 


Is at least one POS currently available in the district? (Probe on why or why not.) 
What is the plan for expanding the district’s POS offerings in future? Have you 
submitted all of your POS to the state for approval? Were any POS submitted by 


the district rejected? (If yes, probe on how many and why.) 


What is your assessment of the general quality of the POS offered in the district? 
(Probe on the characteristics of high-quality POS and the steps that are being taken 


to improve quality if there are concerns in this area.) 
How does a POS differ from a Tech Prep program? 


Assessing Outcomes of POS Implementation: Although states are not required to 
report on POS outcomes, these questions will help us understand how states are 


measuring the effects of these programs. 


Do you have a statewide system or process for tracking POS enrollment? For 


tracking POS outcomes? 


o Ifyes, who maintains it? Is it differentiated by program? Does it contain 


data on both secondary and postsecondary students? 


o Ifno, have you developed a local system for tracking POS enrollment and 


outcomes? 


Do you have the capacity to track or follow secondary POS students into two-year 
postsecondary institutions in the state? Into baccalaureate institutions? Is this ca- 
pacity limited to secondary students who remain in the same POS from secondary 
to postsecondary or those who remain within an established second- 

aty / postsecondary partnership? (Probe on extent of capacity to track secondary to 
postsecondary transitions and ask for summaries or examples of tracking data that 
show extent to which students who begin a POS at secondary level continue at 


postsecondary level.) 


APPENDIX C. CROSS CASE REPORT 


Funding 


DISTRIBUTION OF FUNDING: These questions will help us understand how the exist- 


ing allocation formula supports or inhibits POS development in particular and the attain- 


ment of the Act’s provisions in general. 


95. 


96. 


97. 


98. 


99. 


100. 


101. 


How does the state allocate its Perkins IV dollars for secondary CTE? (Probe on 


details of allocation formula and changes since Perkins III.) 


What are local subgrantees (1.e., districts and schools) required to do to obtain 
funding? (Probe on application process, use of RFPs, use of local plans, and rea- 


sons why a district might not get funding.) 


What is the total (Perkins, state, and district) expenditure on CTE in the district? 


(Probe on proportion from each and changes since Perkins II]. 


How does the district spend its Perkins IV dollars? (Probe on changes in categories 
and percentages since Perkins III.) To what extent does Perkins funding comple- 


ment or supplement state categorical funds? 


What is the district’s mechanism for allocating Perkins IV funds to schools? (Probe 
on proposal and funding processes including use of RFPs or formulas and changes 
since Perkins II.) What percentage of the district’s Perkins IV allocation goes to 


schools? 


Are there other funding strategies that might be adapted to Perkins IV allocations 
in the state to improve CTE programs and outcomes at the district level? (Probe 


for specifics.) 


Have expenditures for nontraditional training and employment changed under Per- 
kins IV? Why ot why not? (Probe on whether district believes continuation of non- 


traditional measures and programs is warranted.) 


Accountability 
ACCOUNTABILITY SYSTEM DESIGN: These questions will help us understand how 


your state and district measure Perkins outcomes. 


102. 


103. 


How does the district define CTE? What definitions does the district use for CTE 
participant and concentrator? (Probe on numbers in each group.) What guidance 


did you receive from the state agency with regard to these definitions? 


How are Perkins IV accountability data collected in the district? (Probe for designa- 
tion of responsibilities [i.e., who does what], types of training and TA provided, 


types of follow up to assure data quality, and changes since Perkins IIL. 
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104. What specific measures are used to collect data about Perkins core indicators in the 
district? What role did the state agency play in the construction of these measures? 
(Probe on why specific measures were chosen and extent to which there is confi- 


dence in the quality of the data.) 


DIFFICULTY IN COLLECTING DATA: These questions are intended to assist us in un- 
derstanding the obstacles states face in collecting data on the measures and specific sub- 


populations of students. 


105. What is the capacity of the district and schools to provide Perkins IV accountability 
data? (Probe on which data are difficult to collect, the level of burden imposed on 


districts, and recommendations for reducing this burden.) 


106. What populations are included in the district’s Perkins reporting system? How 
much confidence do you have in the data that you report on special populations 
and subpopulations (e.g., individuals with disabilities, single parents including 
teen parents, displaced homemakers, economically disadvantaged students in- 
cluding foster children, academically disadvantaged students, English language 
learners, and students pursuing occupations that are nontraditional for their 
gender)? (Probe on capacity to report disaggregated data and suggestions for im- 


provement.) 


107. What investment has the district made in data systems or staffing to meet Perkins 
IV data requirements? (Probe on changes in staff FTEs and resources required to 


build or adapt MIS systems, design surveys, train staff, etc.) 


Program Improvement 
USE OF DATA TO SUPPORT PROGRAM IMPROVEMENT: These questions will help 


us understand how the state and local agencies are using data. 


108. How are accountability data used by the district? By individual schools? 


109. How were the district’s adjusted performance levels on core indicators negotiated 


with the state? (Probe on suggested changes to the negotiation process.) 


110. What is the impact of these performance levels in your district? How have they af- 


fected state and local capacity to identify programs in need of improvement? 


111. Are you aware of rewards and sanctions related to local adjusted performance lev- 
els? (Probe on criteria for threshold levels for state actions, process used, impact 
on local programs.) Have any of your programs been subject to the specific sanc- 
tions outlined in Perkins IV and, if so, have the sanctions led to program improve- 


ment? 
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Other 


112. 


113. 


How is information about CTE communicated to the superintendent, the board of 


education, and the general public? (Ask for examples of reports or agenda items.) 


What are the certification requirements for CTE teachers in the district? Are they 
similar to the certification requirements for other secondary teachers? Is alternative 
licensure an option for CTE teachers? (Probe on challenges associated with finding 


qualified CTE teachers and options for getting them into the classroom. 


Local Postsecondary CTE Director 


Context/Perspective 


114. 


What are your current roles/responsibilities in the implementation of Perkins IV? 
(Probe on changes since Perkins LI, role related to career and technical education 
(CTE) funding, role related to CTE accountability, and role related to programs of 
study [POS].) 


Programs of Study 
POS CREATION: This section will help us to understand how POS were developed within 


the state. See the relevant state survey questions for background. 


115. 


116. 


117. 


118. 


119. 


What progress has the institution made toward developing POS? What is the work- 
ing definition of a POS in your state and how was it developed? How was it com- 


municated to postsecondary institutions? 


What role(s) does the state agency play in the development of POS? Are POS de- 
veloped at the state level, the district/consortia level, or both? Is there a template 


or format that the state uses to guide POS development? 


What role(s) does your institution play in the development of POS? Is POS devel- 
opment in your community typically led by secondary institutions, postsecondary 
institutions, or the two working as partners? How did employers and other busi- 


ness groups participate in the development process? 


How comparable are POS within a given pathway (e.g., construction or health) 
across the state? Are common cutticula and assessments used in similar POS 


actoss the state? 


POS Implementation: These questions will help us understand how implementa- 


tion of POS occurred at both the state and local levels. 
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120. What progress has the institution made toward implementing POS? Specifically, 
what factors have facilitated implementation? What barriers have you encountered? 


(Probe on specific facilitating factors and barriers.) 


121. What kinds of technical assistance does the state provide to postsecondary institu- 
tions for implementation of POS? (Probe for specific examples such as workshops 
ot individual consultations.) To what extent did your institution participate in these 
activities? Did you have sufficient information and support on POS development 


and implementation? 


122. Is at least one POS currently available at the institution? (Probe on why or why 
not.) What is the plan for expanding the institution’s POS offerings in future? Have 
you submitted all of your POS to the state for approval? Were any POS submitted 
by the institution rejected? (If yes, probe on how many and why.) 


123. What is your assessment of the general quality of the POS offered at the institu- 
tion? (Probe on the characteristics of high-quality POS and the steps that are being 


taken to improve quality if there are concerns in this area.) 
124. How does a POS differ from a Tech Prep program? 


ASSESSING OUTCOMES OF POS IMPLEMENTATION: Although states are not re- 
quired to report on POS outcomes, these questions will help us understand how states are 


measuring the effects of these programs. 


125. Do you have a statewide system or process for tracking POS enrollment? For 


tracking POS outcomes? 


oO Ifyes, who maintains it? Is it differentiated by program? Does it contain 


data on both secondary and postsecondary students? 


o Ifno, have you developed a local system for tracking POS enrollment and 


outcomes? 


126. Do you have the capacity to track or follow secondary POS students into two-year 
postsecondary institutions in the state? Into baccalaureate institutions? Is this ca- 
pacity limited to secondary students who remain in the same POS from secondary 
to postsecondary or those who remain within an established second- 
aty/ postsecondary partnership? (Probe on extent of capacity to track secondary to 
postsecondary transitions and ask for summaries or examples of tracking data that 
show extent to which students who begin a POS at secondary level continue at 


postsecondary level.) 
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Funding 


DISTRIBUTION OF FUNDING: These questions will help us understand how the exist- 
ing allocation formula supports or inhibits POS development in particular and the attain- 


ment of the Act’s provisions in general. 


127. How does the state allocate its Perkins IV dollars for postsecondary CTE? (Probe 


on details of allocation formula and changes since Perkins III.) 


128. What are local suberantees (Le., postsecondary institutions) required to do to ob- 
tain funding? (Probe on application process, use of RFPs, use of local plans, and 


reasons why an institution might not get funding.) 


129. What is the total (Perkins, state, and other) expenditure on CTE at the institution? 


(Probe on proportion from each and changes since Perkins III). 


130. How does the institution spend its Perkins IV dollars? (Probe on changes in catego- 
ries and percentages since Perkins III.) To what extent does Perkins funding com- 


plement or supplement state categorical funds? 


131. Are there other funding strategies that might be adapted to Perkins IV allocations 
in the state to improve CTE programs and outcomes at the institution level? 


(Probe for specifics.) 


132. Have expenditures for nontraditional training and employment changed under Per- 
kins [V2 Why ot why not? (Probe on whether institution believes continuation of 


nontraditional measures and programs is warranted.) 


Accountability 


ACCOUNTABILITY SYSTEM DESIGN: These questions will help us understand how 


your state and postsecondary institution measure Perkins outcomes. 


133. How does the institution define CTE? What definitions does the institution use for 
CTE participant and concentrator? (Probe on numbers in each group.) What guid- 


ance did you receive from the state agency with regard to these definitions? 


134. How are Perkins IV accountability data collected at the institution? (Probe for des- 
ignation of responsibilities [i.e., who does what], types of training and TA provided, 


types of follow up to assure data quality, and changes since Perkins LIT.) 


135. What specific measures are used to collect data about Perkins core indicators at the 
institution? What role did the state agency play in the construction of these 
measures? (Probe on why specific measures were chosen and extent to which there 


is confidence in the quality of the data.) 
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DIFFICULTY IN COLLECTING DATA: These questions are intended to assist us in un- 


derstanding the obstacles states face in collecting data on the measures and specific sub- 


populations of students. 


136. 


137. 


138. 


What is the capacity of the institution to provide Perkins IV accountability data? 
(Probe on which data are difficult to collect, the level of burden imposed on post- 


secondary institutions, and recommendations for reducing this burden.) 


What populations are included in the institution’s Perkins reporting system? How 
much confidence do you have in the data that you report on special populations 
and subpopulations (e.g., individuals with disabilities, single parents including teen 
patents, displaced homemakers, economically disadvantaged students including 
foster children, academically disadvantaged students, English language learners, and 
students pursuing occupations that are nontraditional for their gender)? (Probe on 


capacity to report disagereeated data and suggestions for improvement. 
P Pp gereg: ge P 


What investment has the institution made in data systems or staffing to meet Per- 
kins IV data requirements? (Probe on changes in staff FTEs and resources required 


to build or adapt MIS systems, design surveys, train staff, etc.) 


Program Improvement 
USE OF DATA TO SUPPORT PROGRAM IMPROVEMENT: These questions will help 


us understand how the state and local postsecondary institutions are using data. 


Other 


139. 


140. 


141. 


142. 


143. 


How are accountability data used by the institution? 


How were the institution’s adjusted performance levels on core indicators negotiat- 


ed with the state? (Probe on suggested changes to the negotiation process.) 


What is the impact of these performance levels at your institution? How have they 


affected state and local capacity to identify programs in need of improvement? 


Are you aware of rewards and sanctions related to local adjusted performance lev- 
els? (Probe on criteria for threshold levels for state actions, process used, and im- 
pact on local programs.) Have any of your programs been subject to the specific 
sanctions outlined in Perkins IV and, if so, have the sanctions led to program im- 


provement? 


How is information about CTE communicated to community college administra- 
tors, governing bodies (e.g., boards of trustees or governors), and the general pub- 


lic? (Ask for examples of reports or agenda items.) 
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144, What are the employment criteria for CTE instructors at the institution? Are they 


similar to the criteria for other postsecondary instructors? (Probe on challenges as- 
sociated with finding qualified CTE instructors and options for getting them into 


the classroom.) 


Secondary CTE Instructor 


Context/Perspective 


145. What are your current roles/responsibilities in the implementation of Perkins IV? 


Programs of Study 
POS CREATION: This section will help us to understand how POS were developed within 


the state. See the relevant state survey questions for background. 


146. 


147. 


148. 


149. 


150. 


151, 


What is the working definition of a POS in your state? How was it communicated 


to local education agencies (LEAs)? 


Have you or other staff at this school contributed to the development of POS? If 


yes, probe on the nature of the contributions. If no, probe on why not. 


What role(s) does the district play in the development of POS? Is POS develop- 
ment in your community typically led by secondary institutions, postsecondary in- 
stitutions, or the two working as partners? How did employers and other business 


groups participate in the development process? 


POS Implementation: These questions will help us understand how implementa- 


tion of POS occurred at both the state and local levels. 


What progress has the district made toward implementing POS? Specifically, what 
factors have facilitated implementation? What barriers have you encountered? 


(Probe on specific facilitating factors and barriers.) 


Does your district or school offer dual enrollment courses or other opportunities 
for students to earn postsecondary credits at the secondary level? (If yes, probe for 


details on these opportunities and how they’re defined.) 


o For districts that offer opportunities for students to earn postsecond- 
ary credits, ask what courses are offered by whom and what qualifications 


are required to teach these courses. 


oO For districts that don’t offer such opportunities, probe on why not and 


plans to do so in the future. 
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152. 


Funding 


What motivates secondary students to enroll in a POS? Is college credit a motiva- 


tor for secondary students? (Probe on why or why not.) 


DISTRIBUTION OF FUNDING: These questions will help us understand how the exist- 
ing allocation formula supports or inhibits POS development in particular and the attain- 


ment of the Act’s provisions in general. 


153. 


154. 


What is the district’s mechanism for allocating Perkins IV funds to schools? (Probe 
on proposal and funding processes including use of RFPs or formulas and changes 
since Perkins LIT.) 


How are Perkins funds used within your school? How ate decisions made regarding 


use of Perkins funds? 


Accountability 
ACCOUNTABILITY SYSTEM DESIGN: These questions will help us understand how 


your state and district measure Perkins outcomes. 


155. 


156. 


157. 


158. 


159. 


How does the district define CTE? What definitions does the district use for CTE 
participant and concentrator? (Probe on numbers in each group.) What guidance 


did you receive from the state agency regarding these definitions? 


How are Perkins IV accountability data collected in your school? (Probe for desig- 
nation of responsibilities [ie., who does what], types of training and TA provided, 


types of follow up to assure data quality, and changes since Perkins LIL.) 


Difficulty in Collecting Data: These questions are intended to assist us in under- 
standing the obstacles states face in collecting data on the measures and specific 


subpopulations of students. 


What is your school’s capacity to provide Perkins IV accountability data? (Probe on 
which data are difficult to collect, the level of burden imposed on districts, and 


recommendations for reducing this burden.) 


What populations are included in your Perkins reporting system? How much con- 
fidence do you have in the data that you report on special populations and sub- 
populations (e.g., individuals with disabilities, single parents including teen parents, 
displaced homemakers, economically disadvantaged students including foster chil- 
dren, academically disadvantaged students, English language learners, and students 
pursuing occupations that are nontraditional for their gender)? (Probe on capacity 


to report disaggregated data and suggestions for improvement.) 
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Program Improvement 
USE OF DATA TO SUPPORT PROGRAM IMPROVEMENT: These questions will help 


us understand how the state and local agencies are using data. 


160. 


161. 


How ate accountability data used by the school? 


Are you aware of sanctions related to Perkins performance? Have any of your pro- 
y' P y ory 
grams been subject to the specific sanctions related to Perkins performance and, if 


so, have the sanctions led to program improvement? 


Postsecondary CTE Instructor 


Context/Perspective 


162. 


What are your current roles/responsibilities in the implementation of Perkins IV? 


Programs of Study 
POS CREATION: This section will help us to understand how POS were developed within 


the state. See the relevant state survey questions for background. 


163. 


164. 


165. 


166. 


167. 


168. 


What is the working definition of a POS in your state? How was it communicated 


to postsecondary institutions? 


Have you or other staff at this institution contributed to the development of POS? 


If yes, probe on the nature of the contributions. If no, probe on why not. 


What role(s) does the institution play in the development of POS? Is POS devel- 
opment in your community typically led by secondary institutions, postsecondary 
institutions, or the two working as partners? How did employers and other busi- 


ness groups participate in the development process? 


POS Implementation: These questions will help us understand how implementa- 


tion of POS occurred at both the state and local levels. 


What progress has the institution made toward implementing POS? Specifically, 
what factors have facilitated implementation? What barriers have you encountered? 


(Probe on specific facilitating factors and barriers.) 


Does your institution offer dual enrollment courses or other opportunities for stu- 
dents to earn postsecondary credits at the secondary level? (If yes, probe for details 


on these opportunities and how they’re defined.) 


© For institutions that offer opportunities for students to earn postsec- 
ondary credits, ask what courses are offered and what criteria secondary 


schools, teachers, and/or students must meet for such courses. 
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o For institutions that don’t offer such opportunities, probe on why not 


and plans to do so in the future. 


Funding 
DISTRIBUTION OF FUNDING: These questions will help us understand how the exist- 
ing allocation formula supports or inhibits POS development in particular and the attain- 


ment of the Act’s provisions in general. 


169. How ate Perkins funds used within your institution? How are decisions made te- 


garding use of Perkins funds? 


Accountability 
ACCOUNTABILITY SYSTEM DESIGN: These questions will help us understand how 


your state and district measure Perkins outcomes. 


170. How does the institution define CTE? What definitions does the institution use for 
CTE participant and concentrator? (Probe on numbers in each group.) What guid- 


ance did you receive from the state agency with regard to these definitions? 


171. How are Perkins IV accountability data collected at the institution? (Probe for des- 
ignation of responsibilities [i.e., who does what], types of training and TA provided, 


types of follow up to assure data quality, and changes since Perkins LIL.) 


172. Difficulty in Collecting Data: These questions are intended to assist us in under- 
standing the obstacles states face in collecting data on the measures and specific 


subpopulations of students. 


173. What is the capacity of the institution to provide Perkins IV accountability data? 
(Probe on which data are difficult to collect, the level of burden imposed on insti- 


tutions, and recommendations for reducing this burden.) 


174. What populations are included in your Perkins reporting system? How much con- 
fidence do you have in the data that you report on special populations and sub- 
populations (e.g., individuals with disabilities, single parents including teen parents, 
displaced homemakers, economically disadvantaged students including foster chil- 
dren, academically disadvantaged students, English language learners, and students 
pursuing occupations that are nontraditional for their gender)? (Probe on capacity 


to report disaggregated data and suggestions for improvement.) 


Program Improvement 
USE OF DATA TO SUPPORT PROGRAM IMPROVEMENT: These questions will help 


us understand how the state and local agencies are using data. 
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175. How are accountability data used by the institution? 


176. Are you aware of rewards and sanctions related to Perkins performance? Have any 
of your programs been subject to the specific sanctions related to Perkins perfor- 


mance and, if so, have the sanctions led to program improvement? 


APPENDIX C. CROSS CASE REPORT C-155 


Attachment B. Performance Measures 


Perkins IV identifies a set of core indicators to be used by states and local subgrantees to assess pro- 
gram effectiveness. While Perkins IV does not stipulate how sabgrantees should measure performance 
against these indicators, the Department’s nonregulatory guidance includes performance measures 
that identify the proposed numerator and denominator for each core indicator. Attachment B pro- 
vides a comparison of the Department’s recommended performance measures with those adopted at 


the secondary and postsecondary levels in the six case study states. 


Secondary Performance Measures 


ACADEMIC ATTAINMENT—READING/LANGUAGE ARTS 


OVAE Numerator: Number of CTE concentrators who have met the proficient or 
advanced level on the Statewide high school reading/language arts assess- 
ment administered by the State under Section 1111(b)(3) of the Ementary and 
Secondary Education Act (ESEA) as amended by the No Child Left Behind Act 
based on the scores that were included in the State’s computation of adequate 
yearly progress (AYP) and who, in the reporting year, left secondary educa- 
tion. 


guidance 


Denominator: Number of CTE concentrators who took the ESEA assess- 
ments in reading/language arts whose scores were included in the State’s 
computation of AYP and who, in the reporting year, left secondary education. 


State A Number of secondaty CTE completers performing proficient or above on the 
NCLB assessment. 


State B Numerator: Number of CTE concentrators who have met the proficient or 
advanced level on the Statewide high school assessment administered under 
NCLB and who have left secondary education in the reporting year. 


Denominator: Number of CTE concentrators who took the NCLB assess- 
ment and who have left secondary education in the reporting year. 


State C Numerator: Sum of 12th grade CTE concentrators with valid scores 
who have met the proficient or advanced level on the reading/language 
arts portion of the NCLB assessment and who left secondary education 
in the reporting year. 

Denominator: Sum of 12th grade CTE concentrators with valid scores 
on the reading/language arts portion of the NCLB assessment and 
who left secondary education in the reporting year. 


State D Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 
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ACADEMIC ATTAINMENT—READING/LANGUAGE ARTS—CONTINUED 


State F Numerator: Used OVAE guidance. 


Denominator: Used OVAE guidance. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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ACADEMIC ATTAINMENT—MATHEMATICS 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators who have met the proficient or 
advanced level on the Statewide high school mathematics assessment admin- 
istered by the State under Section 1111(b)(3) of the Exmentary and Secondary 
Education Act (ESEA) as amended by the No Child Left Behind Act based on the 
scores that were included in the State’s computation of AYP and who, in the 
reporting year, left secondary education. 


Denominator: Number of CTE concentrators who took the ESEA assess- 
ments in mathematics whose scores were included in the State’s computation 
of AYP and who, in the reporting year, left secondary education 


Number of secondary CTE completers performing proficient or above on the 
NCLB assessment. 


Numerator: Number of CTE concentrators who have met the proficient or 
advanced level on the Statewide high school assessment administered under 
NCLB and who have left secondary education in the reporting year. 


Denominator: Number of CTE concentrators who took the NCLB assess- 
ment and who have left secondary education in the reporting year. 


Numerator: Sum of 12th grade CTE concentrators with valid scores who 
have met the proficient or advanced level on the mathematics portion of 
the NCLB assessment and who left secondary education in the reporting 
year. 

Denominator: Sum of 12th grade CTE concentrators with valid scores on 
the mathematics portion of the NCLB assessment and who left secondary 
education in the reporting year. 

Numerator: Used OVAE guidance. 


Denominator: Used OVAE guidance. 


Numerator: Number of CTE concentrators who have met the proficient or 
advanced level on the Statewide high school mathematics assessment adminis- 
tered by the State under Sec. 1111(b)(3) of ESEA as amended by NCLB based 
on the scores that were included in the State’s computation of AYP. 


Denominator: Number of CTE concentrators who took the ESEA assess- 


ments in mathematics whose scores were included in the State’s computation 
of AYP. 


Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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TECHNICAL SKILL ATTAINMENT 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators who passed technical skill as- 
sessments that are aligned with industry-recognized standards, if available and 
appropriate, during the reporting year. 


Denominator: Number of CTE concentrators who took the assessments dut- 
ing the reporting year. 


Numerator: Lacking statewide skill assessments for all career programs of 
study, the state has determined that the most valid measure of technical 
skill attainment for core indicator 2S1 will be a similar process used under 
Perkins III. Locally determined methods that may include industry certifica- 
tion, third party assessment, locally developed criterion referenced assess- 
ments, ot locally developed methodologies, that may include career and 
technical grade point average, are all options available to schools and depends 
upon the program of study area offered locally. 


Numerator: Number of CTE concentrators who have passed an industry- 
based assessment and who have left secondary education in the reporting year. 


Denominator: Number of CTE concentrators of programs with industry as- 
sessments and who have left secondary education in the reporting year. 


Numerator: Number of secondary CTE concentrators who passed an end-of- 
program technical skill assessment that is aligned with industry-recognized 
standards, including the State CTE model curriculum standards, during the 
reporting year. 


Denominator: Number of secondary CTE concentrators who took an end-of- 
program technical skill assessment during the reporting year. 


Numerator: Number of senior CTE concentrators who (1) earned an industry 
certified credential through a third-party assessment or (2) successfully passed 
a State-approved end-of-course or end-of program assessment. 


Denominator: Number of senior CTE concentrators who have left secondary 
education in the reporting year. 


Numerator: Number of CTE concentrators who passed state 

recognized technical skill assessments that are aligned with 
industry-recognized standards, if available and appropriate, 

during the reporting year. 

Denominator: Number of CTE concentrators who took the state 
recognized assessments during the reporting year. 

Numerator: Number of 12th-grade concentrators who have mastered indus- 
try-validated career and technical proficiency standards in the reporting year. 


Denominator: Total number of 12th-grade concentrators who have left the 
system in the reporting year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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SECONDARY SCHOOL COMPLETION 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators who earned a regular secondary 
school diploma; earned a General Education Development (GED) credential 
as a State-recognized equivalent to a regular high school diploma (if offered by 
the State) or other State-recognized equivalent (including recognized alterna- 
tive standards for individuals with disabilities), or earned a proficiency creden- 
tial, certificate, or degree, in conjunction with a secondary school diploma (if 
offered by the State) during the reporting year. 


Denominator: Number of CTE concentrators who left secondary education 
during the reporting year. 


Numerator: Number of 12th-grade CTE program concentrators earning a 
high school diploma by June 30. 


Denominator: Number of 12th-grade CTE program concentrators for the 
program year ending on June 30. 


Numerator: Number of CTE concentrators who have attained a high school 
diploma or GED certificate and who have left secondary education in the re- 
porting year. 


Denominator: Used OVAE guidance. 


Numerator: Number of 12th-grade CTE program completers earning a high 
school diploma by June 30. 


Denominator: Number of 12th-grade CTE program completers for the pro- 
gram year ending on June 30. 


Numerator: Number of senior CTE concentrators who (1) attained a stand- 
atd high school diploma, (2) GED credential or Adult High School diploma, 
ot (3) a proficiency credential, certificate, or degree in conjunction with a sec- 
ondary school diploma. 


Denominator: Number of senior CTE concentrators who have left secondary 
education in the reporting year. 


Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


Numerator: Number of 12th-grade concentrators who attained a high school 
diploma, a State certificate, or a GED. 


Denominator: Number of 12th-grade concentrators who have left the system 
in the reporting year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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STUDENT GRADUATION RATES 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators who, in the reporting year, were 
included as graduated in the State’s computation of its graduation rate as de- 
sctibed in Section 1111(b)(2)(C)(vi) of the ESEA. 


Denominator: Number of CTE concentrators who, in the reporting year, 
were included in the State's computation of its graduation rate as defined in the 
State’s Consolidated Accountability Plan pursuant to Section 1111(b)(2)(C)(vi) 
Out Wave JESIEVAL 


Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


Numerator: Number of concentrators reported as graduated using our State’s 
approved calculation for graduation rate as defined in our State’s NCLB ac- 
countability workbook. 


Denominator: Number of concentrators who have left secondary education 
in the reporting year. 


Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 
Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 
Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


Numerator: Used OVAE guidance. 


Denominator: Used OVAE guidance. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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SECONDARY PLACEMENT 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


Numerator: Number of CTE concentrators who left secondary education and 
were placed in postsecondary education or advanced training, in the military 
setvice, ot employment in the second quarter following the program year in 
which they left secondary education (i.e., unduplicated placement status for 
CTE concentrators who graduated by June 30, 2007 would be assessed be- 
tween October 1, 2007 and December 31, 2007). 


Denominator: Number of CTE concentrators who left secondary education 
during the reporting year. 


Numerator: Number of the 12th-grade CTE program completers in the mili- 
tary, enrolled in further education or training, or employed six months after the 
program year exited from high school. 


Denominator: Total number of the 12th-grade CTE program completers who 
exited the high school. 


Numerator: Number of CTE concentrators who were employed, enrolled in 
higher education, or enlisted in the military during the third post-exit quarter, 
based on administrative records or a student sutvey. 


Denominator: Used OVAE guidance. 


Numerator: The number of the 12th-grade CTE program completers in the 
military, enrolled in further education or training, or employed six months after 
exiting the high school. 


Denominator: The total number of the 12th-grade program completers who 
exited the high school. 


Numerator: Number of CTE concentrators from the prior year who com- 
pleted secondary school and who were placed in postsecondary education, 
employment, and /or military service in the 2nd quarter (October-December) 
after leaving secondary education during the report year. 


Denominator: Number of CTE concentrators from the prior yeat who com- 
pleted secondary school. 


Numerator: Number of known status CTE concentrators who left secondary 
education the previous school year and were identified as placed in postsec- 
ondary education or advanced training, in the military service, or employment 
in the second quarter following the program year in which they left secondary 
education (i.e., unduplicated placement status for CTE concentrators who left 
secondary education |i.e. graduated or withdrew] by June 30, 2007 and were 
identified by local survey as in placement between October 1, 2007 and De- 
cember 31, 2007). 


Denominator: Number of known CTE concentrators who left secondary ed- 
ucation during the previous school year. 
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SECONDARY PLACEMENT—CONTINUED 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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NONTRADITIONAL PARTICIPATION 


State A Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


State C Numerator: Sum of CTE participants from underrepresented gender 
groups enrolled in a program sequence that leads to employment in non- 

traditional fields during the reporting year. 

Denominator: Sum of CTE participants enrolled in a program sequence 

that leads to employment in nontraditional fields during the reporting year. 


State E Numerator: Number of CTE participants from underrepresented gender 
groups who participated in a program that, as specified by the National As- 
sessment of Educational Progress (NAPE), leads to employment in nontradi- 
tional fields during the reporting year. 


Denominator: Number of CTE participants who participated in a program 
that, as specified by NAPE, leads to employment in nontraditional fields 
during the reporting year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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NONTRADITIONAL COMPLETION 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators from underrepresented gender 
groups who completed a program that leads to employment in nontraditional 
fields during the reporting year. 


Denominator: Number of CTE concentrators who completed a program that 
leads to employment in nontraditional fields during the reporting year. 


The completion rate is calculated in the same way as the completion rate of all 
students, but only using the students enrolled in nontraditional programs of 
study. 


Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


Numerator: Sum of CTE concentrators from underrepresented gender 
groups who completed a program sequence that leads to employment in 
nontraditional fields during the reported year. 

Denominator: Sum of all CTE concentrators from underrepresented 
gender groups enrolled in a program sequence that leads to employment 

in nontraditional fields during the reporting year. 

Numerator: Number of senior CTE concentrators in programs identified as 
nontraditional for their gender who (1) attained a standard high school diplo- 
ma, (2) GED credential or Adult High School diploma, or (3) a proficiency 
credential, certificate, or degree in conjunction with a secondary school diplo- 
ma. 


Denominator: Number of senior CTE concentrators in programs identified 
as nontraditional for their gender that have left secondary education in the re- 
porting year. 


Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


Numerator: Number of CTE concentrators from underrepresented gender 
groups who participated in a course that leads to employment in nontraditional 
fields in the reporting year. 


Denominator: Number of CTE concentrators who participated in a course 
that leads to employment in nontraditional fields in the reporting year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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Postsecondary Performance Measures 


TECHNICAL SKILL ATTAINMENT 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators who passed technical skill as- 
sessments that are aligned with industry-recognized standards, if available and 
appropriate, during the reporting year. 


Denominator: Number of CTE concentrators who took the assessments dut- 
ing the reporting year. 


The percentage of students earning a GPA of 2.0 or higher in CTE courses. 


Number of CTE concentrators, exiting during the reporting year, who have 
attained an award (a degtee, certificate, apprenticeship, or an industry certifica- 
tion) or completed at least 45 vocational credits with a 2.0 or higher GPA. 


The State will use the existing approved Perkins III technical skill attainment 
measure as authorized by Sec. 113(b)(2)(D) of Perkins IV as the percentage of 
students earning a GPA of 2.0 or higher in CTE courses. 


Numerator: Number of CTE concentrators who (1) earned an industry certi- 
fied credential through a third-party assessment or (2) earned 75 percent of the 
program houts required with a GPA of 2.5 or higher. 


Denominator: Number of CTE concentrators in the reporting year. 


Numerator: Number of CTE concentrators who passed state recognized 
technical skill assessments that are aligned with industry-recognized stand- 
ards, if available and appropriate, during the reporting year. 

Denominator: Number of CTE concentrators who took state recognized 
technical skill assessments during the reporting year. 

Numerator: During the reporting year, the number of CTE completers who 
passed, on the first administration, major field assessments that are aligned 
with industry-recognized standards, if available and appropriate. 


Denominator: Number of CTE completers who took major field assessments 
for the first time during the reporting year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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CREDENTIAL, CERTIFICATE, OR DIPLOMA 


State A The percentage of concentrators who have successfully completed a minimum 
“threshold of 12 or more units of related coursework” in a CTE program area 
ot received a vocational certificate of less than 12 units and who (1) receive a 
degtee, certificate, or equivalent or (2) complete a transfer program and are 
classified as transfer ready. 


State C The percentage of “Leavers and Completers” who have successfully completed 
a minimum “threshold of 12 or more units of related coursework” in a CTE 
program area or received a vocational certificate of less than 12 units and who 
(1) receive a degree, certificate, or equivalent or (2) complete a transfer pro- 
eram and are classified as “transfer prepared.” 


State E Numerator: Used OVAE guidance. 


Denominator: Number of CTE concentrators who graduated or withdrew 
from postsecondary education during the reporting year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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STUDENT RETENTION OR TRANSFER 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators who remained enrolled in their 
original postsecondary institution or transferred to another 2- or 4-year post- 
secondary institution during the reporting year and who were enrolled in post- 
secondary education in the fall of the previous reporting year. 


Denominator: Number of CTE concentrators who were enrolled in postsec- 
ondary education in the fall of the previous reporting year and who did not 
earn an industry-recognized credential, a certificate, or a degree in the previous 
reporting year. 


Percentage of CTE student concentrators who have successfully completed a 
minimum “threshold of 12 or more units of related coursework” in a CTE 
program area and who (1) persisted in education at the community college lev- 
el or (2) transferred to a four-year institution. Work continues to define the 
parameters for determining the measure of persistence. 


Numerator: Number of CTE participants who are not yet concentrators at 
the beginning of the reporting year, who became CTE concentrators or en- 
rolled in other higher education, including apprenticeship, during the reporting 
year. 


Denominator: Number of CTE participants during the reporting year who 
are not yet concentrators at the beginning of the reporting year. 


The percentage of CTE student concentrators who have successfully complet- 
ed a minimum “threshold of 12 or more units of related coursework” in a 
CTE program area and who (1) persisted in education at the community col- 
lege level or (2) transferred to a four-year institution. 


Numerator: Number of CTE concentrators who remained enrolled in their 
original postsecondary institution or transferred to another two- or four-year 
postsecondary institution during the reporting year and who were enrolled in 
postsecondary education in the previous reporting year. 


Denominator: Number of CTE concentrators from the prior year who did 
not earn an industry-recognized credential, a certificate, or a degree in the pre- 
vious reporting year. 


Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


Numerator: Number of CTE concentrators who remained enrolled in their 
original community college, completed a degree or award at their original 
community college, or transferred to another two- or four-year postsecondary 
institution at the time of the subsequent fall after the sophomore concentrator 
year. 


Denominator: Number of CTE concentrators in the fall of the sophomore 
cohott yeat. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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STUDENT PLACEMENT 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators who were placed or retained in 
employment, or placed in military service or apprenticeship programs in the 
2nd quarter following the program year in which they left postsecondary educa- 
tion (i.e., unduplicated placement status for CTE concentrators who graduated 
by June 30, 2007 would be assessed between October 1, 2007 and December 
31, 2007). 


Denominator: Number of CTE concentrators who left postsecondary educa- 
tion during the reporting year. 


The final agreed upon 2008-09 performance level for this core indicator will 
be determined after April 1, 2008 through a process of negotiations with the 
U.S. Department of Education. 


Numerator: Number of CTE concentrators, who were either employed ac- 
cording to UI wage records or in the military, and not enrolled in higher educa- 
tion during the third quarter after they exit. 


Denominator: Number of CTE concentrators exiting during the reporting 
period and not enrolled in higher education during the 3rd quarter after exit. 


The percentage of CTE program leavers and completers who did not transfer 
to a two- or four-year institution and were found during one of the four quar- 
ters following the cohort year in an apprenticeship program, UI-covered em- 
ployment, the federal government, or the military. 


Numerator: Number of CTE concentrators from the prior year who received 
a credential, degree, or certificate who were placed in postsecondary education, 
employment, and/or military service in the second quarter (October— 
December) after leaving postsecondary education. 


Denominator: Number of CTE concentrators from the prior year who re- 
ceived a credential, degree, or certificate. 


Numerator: Number of CTE concentrators who were placed or retained in 
employment or placed in military service or apprenticeship programs in the 
second quarter following the program year in which they left your postsecond- 
ary institution. 

Denominator: Number of CTE concentrators who graduated or withdrew 


from postsecondary education during the reporting year. 


Numerator: Number of CTE concentrators who received an industry recog- 
nized degree or award and were placed in employment, enrolled in post- 
secondary education, or served in the military. 


Denominator: Number of CTE concentrators who graduated from the com- 
munity colleges at the end of the reporting year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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NONTRADITIONAL PARTICIPATION 


OVAE Numerator: Number of CTE participants from underrepresented gender 
groups who participated in a program that leads to employment in nontradi- 
tional fields during the reporting year. 


guidance 


Denominator: Number of CTE participants who participated in a program 
that leads to employment in nontraditional fields during the reporting year. 


State A Not available. 


State B Numerator: Number of CTE participants from underrepresented gender 
groups who enrolled in a nontraditional program during the reporting period. 


Denominator: Number of CTE participants in nontraditional programs dur- 
ing the reporting period. 


State C The percentage of female concentrators participating in CTE program course- 
work leading to employment in occupations nontraditional for females and 
male concentrators participating in CTE program coursework leading to em- 
ployment in occupations nontraditional for males. 


State D Numerator: Number of CTE participants from underrepresented gender 
groups in college credit programs that lead to employment in nontraditional 
fields during the reporting year. 


Denominator: Number of CTE participants in college credit programs that 
lead to employment in nontraditional fields during the reporting year. 


State E Numerator: Number of CTE participants from underrepresented gender 
groups who participated in a program that, as specified by NAPE, leads to 
employment in nontraditional fields during the reporting year. 


Denominator: Number of CTE participants who participated in a program 
that, as specified by NAPE, leads to employment in nontraditional fields dur- 
ing the reporting year. 


State F Numerator: Used OVAE guidance. 
Denominator: Used OVAE guidance. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 
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NONTRADITIONAL COMPLETION 


OVAE 


guidance 


State A 


State B 


State C 


State D 


State E 


State F 


Numerator: Number of CTE concentrators from underrepresented gender 
groups who completed a program that leads to employment in nontraditional 
fields during the reporting year. 


Denominator: Number of CTE concentrators who completed a program that 
leads to employment in nontraditional fields during the reporting year. 


Not available. 


Numerator: Number of CTE completers from underrepresented gender 
groups who enrolled in a nontraditional program during the reporting period. 
Denominator: Number of CTE completers in nontraditional programs during 
the reporting period. 


Percentage of completers in programs leading to employment in nontraditional 
occupations that are of the underrepresented gender (i.e., female students 
completing programs leading to employment in occupations nontraditional for 
females and male students completing programs leading to employment in 
occupations nontraditional for males). Completion is defined as (1) receiving a 
degree, certificate, or equivalent; (2) completing a transfer program and being 
designated as “transfer-prepared”’; (3) transferring to a two- or four-year insti- 
tution; or (4) enlisting in the military. 


Numerator: Number of CTE concentrators who received an industty- 
recognized credential, certificate, or degree in college credit programs identi- 
fied as nontraditional for their gender. 


Denominator: Number of CTE concentrators in college credit programs 
identified as nontraditional for their gender. 


Numerator: Number of CTE concentrators from underrepresented gender 
groups who graduated/completed a program that, as specified by NAPE, leads 
to employment in nontraditional fields during the reporting year. 


Denominator: Number of CTE concentrators who completed a program 
that, as specified by NAPE, leads to employment in nontraditional fields dur- 
ing the reporting year. 


Numerator: Number of CTE concentrators from underrepresented gender 
groups who completed a program that leads to employment in nontraditional 
fields by the end of two years subsequent to the fall of the sophomore cohort 
year. 


Denominator: Number of CTE concentrators from underrepresented gender 
groups who participated in a program that leads to employment in nontradi- 
tional fields in the fall of the sophomore cohort year. 


SOURCE: Justesen (2007a) and Perkins Five-Year State Plans. 


